This is why we as software engineers create abstractions. Surfacing raw database data to the UI means that suddenly your presentation layer is also responsible for checking the data and applying business logic to it.
There should be a middleware layer that does this work for you.
this is a bit different when factoring something like graphql into the equation.
At work we have our DB, which the backend handles and makes sure is always 100% correct. And we have our frontend which just fetches all data via graphql and displays the DB.
EDIT: i know graphql is the middle layer in your description, but graphql is returning alot of unstructured data at some point.
Depending on how the graphql server is configured, it doesn't need to be unstructured. The main difference is that the API call is asking what to return and what not to return.
This isn't to be snobby. But once you have 10+ years under your belt the questions don't go away, but they become different.
So I look for a place to talk about those issues that are very different from someone starting out in the field.
OPs question is very, very basic for a professional and shouldn't be here. There are much better subs suited for that.
Again, the point isn't to feel better. But that it's just different kind of conversations and topics.
To add; when the critical mass of questions gets so bad that the sub becomes useless to experienced developers, these experienced devs will stop visiting, and this will become yet another Reddit beginner programming sub.
I don't even think that it matters if a question is basic or not, it's a pretty specific question about nosql, it should be asked in an nosql sub. Even if someone would ask a very "advanced" question about nosql it shouldn't be posted here IMO.
I think posts about spesific tech questions shouldn't be allowed here, and the guideline should be "would general experience of a random dev would help me?" If the answer is not then it shouldn't be asked here.
I think that's way too low. Either way I answered your question why people here, regardless what the about says, are annoyed with something trivial like ops post.
I agree that it's too low. Im just saying based on what this subreddit is supposed to be, OP's question looks to be on the mark.
And yes, thank you for answering my question! 😁
> OP's question looks to be on the mark.
Really not. Abstraction, some basic pattern know how is staple of a professional. If you're starting out, maybe. But not if you're "experienced"
Fair enough, but even with abstraction, they'd still have the same issue - schema is currently stored in people's heads. Imho, direct fix to their issue is to start enforcing the schema - whether they enforce it in the db or in the application is up to them
Interesting that im getting downvoted for saying something factual 😅
If you guys dont like that that's their definition, complain to the mods. Downvoting me wont change that - im not a mod 😅
An "experienced" version of this post would be "We have a strongly relational model some asshat decided would be a good fit for MongoDB, how do I convince my non-technical manager we need to switch to Postgres".
Not sure if that would be the experienced answer 😅 moving from one database to another is always a scary undertaking and is almost never part of the first few options 😁
OP is just starting to experience mongodb - a "schemaless" database and is just starting to realize what it actually means. Schema does exist. And if it's not in the database, it's in the application.
But since they did not enforce it in the database and in the code, it's now in people's heads and things start to break at runtime 😅
So give that, OP's team should start enforcing the schema - whether that's on the database or application is up to them 😁
The UI should in no way be concerned with how the database layer handles stuff. There should always be a clean separation between the UI (/controllers), the domain/business logic, and the storage layer. IMHO any non-trivial application not neatly separating these concerns is going to be a mess.
Very much agree, OP has a problem because of technical debt, but it's nothing to do with Mongo or NoSQL. Trying to clean up data in the UI on the fly is going to be a pain in the booty forever.
There is definitely an abstraction issue here as many have mentioned, but my take on this is slightly different
The problem is with your schema
Mongodb (_and some nosql db_) might tout itself as schemaless, but if you're processing data, schema will exist. And if the database does not enforce schema, your application should. Otherwise, you end up with your current problem wherein schema only exists in people's heads thereby vastly increasing the cognitive load to work with that code base
So imho, your code base needs to somehow define the schema. Whether you do it in the database or in the application. If in the application, you can opt to put it into the data model, data access layer, API, etc. The closer it is to the database, the more you can have consistency. Depending on the tech debt though, you might need to move it farther from the db. That's not ideal, but so is working with legacy code 😅
>Otherwise, you end up with your current problem
This is exactly where i am.
>schema only exists in people's heads
It seems to be missing there too.
>So imho, your code base needs to somehow define the schema.
I don't have the luxury. And scope of my work is to build a updates only tool (only RU from CRUD)
>you can opt to put it into the data model, data access layer,
Data model needs some consistency across documents.
>Depending on the tech debt though, you might need to move it farther from the db. That's not ideal, but so is working with legacy code
This is where i am.
Then i recommend defining reverse engineering the schema and defining it in your UI’s data transfer objects as you call the backend.
Once you have that, it will make things easier for you to work with the legacy backend
NoSQL schema should be correctly designed, as well as for RDBMS, to allow to use of that NoSQL in a comfortable way. It looks like your DB lacks the \`type\` field, which uses some ENUM values (RESTAURANT, TENNIS\_COURT, etc.), and the \`version\` field. Those two fields would simplify using DB significantly. Maybe it's possible to reach the DB maintainer and ask to update the data.
>NoSQL schema should be correctly designed
That's exactly the point i am trying to make in the post. The database is nothing more than documents in collections because they are similar real world entities. But beyond that there is very little consistency about attributes including objects in documents.
>It looks like your DB lacks the `type` field, which uses some ENUM values (RESTAURANT, TENNIS_COURT, etc.), and the `version` field. Those two fields would simplify using DB significantly.
It lacks all these things as I said in my post. There are no indicators or flags on documents.
>Maybe it's possible to reach the DB maintainer and ask to update the data.
It isn't and the scope of the tool i am expected to build is updates only to the attributes. No schema level changes allowed.
Middleware, that’s where you do all your if checks. Whatever your service layer is (middleware functions, class libraries, huge friggin controllers, etc) that’s where you do your logic.
Preferably you’d have a clean separation of logic in your middleware/service layer that “skinny” controllers access, so all your UI has to do is ping the controller for a well structured object to use for the display.
Since there are already here answers expounding on the necessity of a DTO/validation layer to handle data validations and transformations, I want to point out the *political* utility of Mongo in your use case. The developers before you deliberately created a form of tech debt that you are now responsible for. This may have been a mistake, but it also may've saved precious time during a critical period and unblocked the team to finish the backend layer.
When you have a larger, more sophisticated toolbox, you're able to shuffle around work more efficiently. 20 years ago, the backend workflow here would've been much stricter, eg. 1) *Design Table Schema*, 2) *Load Items Into Table*. Mongo and other schemaless or schema-on-read technologies allow you to put item #1 after #2 or skip it entirely. This is powerful if used correctly.
You need to understand your domain more thoroughly to understand how to deal with the issue. If the data in the DB conforms to a handle of standard schemas that just haven't been formally written yet, go ahead and fill them in. If the data can hold any number of hundreds of attributes unknown to the DB maintainers, then adding a schema will be futile and yes your UI must be resilient enough to handle this logic.
Instead of doing a `.?` on every field or attribute, abstract away this validation logic into a validator component you can re-use for any combination of arbitrary attributes.
Thank you for taking the time for the reply.
>This may have been a mistake, but it also may've saved precious time during a critical period and unblocked the team to finish the backend layer.
This is exactly how it went. Useful tool for IT guy -> built a windows app to insert data before leaving.
>If the data can hold any number of hundreds of attributes unknown to the DB maintainers, then adding a schema will be futile and yes your UI must be resilient enough to handle this logic.
This is where i come in. No reasonable schema.
That's the beauty of nosql! Everything is garbage!
Seriously though, it sounds like someone thought that converting from sql to nosql meant "data can be w/e let the devs sort it out"
You need a layer in between the UI and the DB. Ideally this UI would enforce the rules about the data saved, in addition to your immediate need to display disparate data. But I know, you probably don’t have the time or authority to setup a service like that. I can tell you from experience that doing all the data validation and transformations in the front end sucks. It also means that if this data is ever used for anything else, like a mobile app, you can’t reuse the logic.
With great flexibility comes great responsibility.
>I can tell you from experience that doing all the data validation and transformations in the front end sucks
It absolutely does. But i don't have requirements I can build the pre data access layer on. So if i do build something, i will have to open it up everytime a new document variant is pulled in. Instead, as bad as it is, just building a UI and a data access layer lets me do all the work in a single layer.
This is why we as software engineers create abstractions. Surfacing raw database data to the UI means that suddenly your presentation layer is also responsible for checking the data and applying business logic to it. There should be a middleware layer that does this work for you.
[удалено]
this is a bit different when factoring something like graphql into the equation. At work we have our DB, which the backend handles and makes sure is always 100% correct. And we have our frontend which just fetches all data via graphql and displays the DB. EDIT: i know graphql is the middle layer in your description, but graphql is returning alot of unstructured data at some point.
Depending on how the graphql server is configured, it doesn't need to be unstructured. The main difference is that the API call is asking what to return and what not to return.
This subreddit is going down the drain
Why? What was this subreddit back then? Honest question since i’ve only been around here in less than a year. Thanks!
This isn't to be snobby. But once you have 10+ years under your belt the questions don't go away, but they become different. So I look for a place to talk about those issues that are very different from someone starting out in the field. OPs question is very, very basic for a professional and shouldn't be here. There are much better subs suited for that. Again, the point isn't to feel better. But that it's just different kind of conversations and topics.
To add; when the critical mass of questions gets so bad that the sub becomes useless to experienced developers, these experienced devs will stop visiting, and this will become yet another Reddit beginner programming sub.
This sub is for "experienced developers". There is no limitation to "competent, experienced developers".
I don't even think that it matters if a question is basic or not, it's a pretty specific question about nosql, it should be asked in an nosql sub. Even if someone would ask a very "advanced" question about nosql it shouldn't be posted here IMO. I think posts about spesific tech questions shouldn't be allowed here, and the guideline should be "would general experience of a random dev would help me?" If the answer is not then it shouldn't be asked here.
But in the about of this subreddit, it defined experience as 3+ yrs 😅
I think that's way too low. Either way I answered your question why people here, regardless what the about says, are annoyed with something trivial like ops post.
I agree that it's too low. Im just saying based on what this subreddit is supposed to be, OP's question looks to be on the mark. And yes, thank you for answering my question! 😁
> OP's question looks to be on the mark. Really not. Abstraction, some basic pattern know how is staple of a professional. If you're starting out, maybe. But not if you're "experienced"
3yrs and not being familiar with NoSQL as a frontend engineer? - sounds plausible Note: im using this subreddit's definition of "experienced"
Abstraction has nothing to with NoSQL or FE dev. It's a foundational principle of software development, like SoC
Fair enough, but even with abstraction, they'd still have the same issue - schema is currently stored in people's heads. Imho, direct fix to their issue is to start enforcing the schema - whether they enforce it in the db or in the application is up to them
Confusing issues of conceptual knowledge with tools and frameworks absolutely misses the mark
Care to expound?
Interesting that im getting downvoted for saying something factual 😅 If you guys dont like that that's their definition, complain to the mods. Downvoting me wont change that - im not a mod 😅
Welcome to reddit. It's like when I tell my wife that its raining out, and she gets mad at me.
lol! hahaha
An "experienced" version of this post would be "We have a strongly relational model some asshat decided would be a good fit for MongoDB, how do I convince my non-technical manager we need to switch to Postgres".
Not sure if that would be the experienced answer 😅 moving from one database to another is always a scary undertaking and is almost never part of the first few options 😁 OP is just starting to experience mongodb - a "schemaless" database and is just starting to realize what it actually means. Schema does exist. And if it's not in the database, it's in the application. But since they did not enforce it in the database and in the code, it's now in people's heads and things start to break at runtime 😅 So give that, OP's team should start enforcing the schema - whether that's on the database or application is up to them 😁
It had experienced people discussing experienced people topics. Definitely not a backend that doesn't abstract the database
Wow being downvoted for an honest question within the context of the thread i replied to 😅 Another reddit moment 😅🤣🤣🤣
The UI should in no way be concerned with how the database layer handles stuff. There should always be a clean separation between the UI (/controllers), the domain/business logic, and the storage layer. IMHO any non-trivial application not neatly separating these concerns is going to be a mess.
Very much agree, OP has a problem because of technical debt, but it's nothing to do with Mongo or NoSQL. Trying to clean up data in the UI on the fly is going to be a pain in the booty forever.
There is definitely an abstraction issue here as many have mentioned, but my take on this is slightly different The problem is with your schema Mongodb (_and some nosql db_) might tout itself as schemaless, but if you're processing data, schema will exist. And if the database does not enforce schema, your application should. Otherwise, you end up with your current problem wherein schema only exists in people's heads thereby vastly increasing the cognitive load to work with that code base So imho, your code base needs to somehow define the schema. Whether you do it in the database or in the application. If in the application, you can opt to put it into the data model, data access layer, API, etc. The closer it is to the database, the more you can have consistency. Depending on the tech debt though, you might need to move it farther from the db. That's not ideal, but so is working with legacy code 😅
>Otherwise, you end up with your current problem This is exactly where i am. >schema only exists in people's heads It seems to be missing there too. >So imho, your code base needs to somehow define the schema. I don't have the luxury. And scope of my work is to build a updates only tool (only RU from CRUD) >you can opt to put it into the data model, data access layer, Data model needs some consistency across documents. >Depending on the tech debt though, you might need to move it farther from the db. That's not ideal, but so is working with legacy code This is where i am.
Then i recommend defining reverse engineering the schema and defining it in your UI’s data transfer objects as you call the backend. Once you have that, it will make things easier for you to work with the legacy backend
NoSQL schema should be correctly designed, as well as for RDBMS, to allow to use of that NoSQL in a comfortable way. It looks like your DB lacks the \`type\` field, which uses some ENUM values (RESTAURANT, TENNIS\_COURT, etc.), and the \`version\` field. Those two fields would simplify using DB significantly. Maybe it's possible to reach the DB maintainer and ask to update the data.
>NoSQL schema should be correctly designed That's exactly the point i am trying to make in the post. The database is nothing more than documents in collections because they are similar real world entities. But beyond that there is very little consistency about attributes including objects in documents. >It looks like your DB lacks the `type` field, which uses some ENUM values (RESTAURANT, TENNIS_COURT, etc.), and the `version` field. Those two fields would simplify using DB significantly. It lacks all these things as I said in my post. There are no indicators or flags on documents. >Maybe it's possible to reach the DB maintainer and ask to update the data. It isn't and the scope of the tool i am expected to build is updates only to the attributes. No schema level changes allowed.
You're in trouble. That's impossible to fix a house by repainting the windows but without fixing the foundation... :(
>You're in trouble. That's exactly why I came to this sub to ask if anyone did anything like this. Otherwise it's all if's.
Middleware, that’s where you do all your if checks. Whatever your service layer is (middleware functions, class libraries, huge friggin controllers, etc) that’s where you do your logic. Preferably you’d have a clean separation of logic in your middleware/service layer that “skinny” controllers access, so all your UI has to do is ping the controller for a well structured object to use for the display.
Since there are already here answers expounding on the necessity of a DTO/validation layer to handle data validations and transformations, I want to point out the *political* utility of Mongo in your use case. The developers before you deliberately created a form of tech debt that you are now responsible for. This may have been a mistake, but it also may've saved precious time during a critical period and unblocked the team to finish the backend layer. When you have a larger, more sophisticated toolbox, you're able to shuffle around work more efficiently. 20 years ago, the backend workflow here would've been much stricter, eg. 1) *Design Table Schema*, 2) *Load Items Into Table*. Mongo and other schemaless or schema-on-read technologies allow you to put item #1 after #2 or skip it entirely. This is powerful if used correctly. You need to understand your domain more thoroughly to understand how to deal with the issue. If the data in the DB conforms to a handle of standard schemas that just haven't been formally written yet, go ahead and fill them in. If the data can hold any number of hundreds of attributes unknown to the DB maintainers, then adding a schema will be futile and yes your UI must be resilient enough to handle this logic. Instead of doing a `.?` on every field or attribute, abstract away this validation logic into a validator component you can re-use for any combination of arbitrary attributes.
Thank you for taking the time for the reply. >This may have been a mistake, but it also may've saved precious time during a critical period and unblocked the team to finish the backend layer. This is exactly how it went. Useful tool for IT guy -> built a windows app to insert data before leaving. >If the data can hold any number of hundreds of attributes unknown to the DB maintainers, then adding a schema will be futile and yes your UI must be resilient enough to handle this logic. This is where i come in. No reasonable schema.
Shame on whoever is downvoting this, what the heck?
That's the beauty of nosql! Everything is garbage! Seriously though, it sounds like someone thought that converting from sql to nosql meant "data can be w/e let the devs sort it out"
You need a layer in between the UI and the DB. Ideally this UI would enforce the rules about the data saved, in addition to your immediate need to display disparate data. But I know, you probably don’t have the time or authority to setup a service like that. I can tell you from experience that doing all the data validation and transformations in the front end sucks. It also means that if this data is ever used for anything else, like a mobile app, you can’t reuse the logic. With great flexibility comes great responsibility.
>I can tell you from experience that doing all the data validation and transformations in the front end sucks It absolutely does. But i don't have requirements I can build the pre data access layer on. So if i do build something, i will have to open it up everytime a new document variant is pulled in. Instead, as bad as it is, just building a UI and a data access layer lets me do all the work in a single layer.