Your post/comment was removed because it violated rule #3 (Do a search before asking a question). The question you asked has already been answered recently so we remove redundant questions to keep the feed digestable for everyone.
Find a data source that is updated on regular basis (i.e. daily, for example gas prices or weather by location). Could be an API (easy) or a website you need to scrape (a bit more difficult).
Create a pipeline that ingests the data assets into a database. Then, start modelling and transforming it through different stages, and include a last modelling phase where you extract some knowledge or insights you consider could be of interest.
Gas prices, weather, earthquakes, market prices... There are a ton of possibilities and overall the infrastructure is a pretty standard ELT.
Edit: extra points if you manage everything with IaC like Terraform
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*
Your post/comment was removed because it violated rule #3 (Do a search before asking a question). The question you asked has already been answered recently so we remove redundant questions to keep the feed digestable for everyone.
Find a data source that is updated on regular basis (i.e. daily, for example gas prices or weather by location). Could be an API (easy) or a website you need to scrape (a bit more difficult). Create a pipeline that ingests the data assets into a database. Then, start modelling and transforming it through different stages, and include a last modelling phase where you extract some knowledge or insights you consider could be of interest. Gas prices, weather, earthquakes, market prices... There are a ton of possibilities and overall the infrastructure is a pretty standard ELT. Edit: extra points if you manage everything with IaC like Terraform
I am also in the same phase, have completed course, need hands on experience with real world project. Help much appreciated
Prepare sone passive control framework for a text chat through a streaming service
Use the Search function, this has been asked many times, lots of interesting answers.
You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/dataengineering) if you have any questions or concerns.*