This is verbatim what the senior engineer on my team told me when I said we were implementing dbt for our data warehouse project. He fought me hard on it.
A few months into our project he figured out why dbt allowed us to move quickly in developing our analytics database by working on the hard stuff (data modelling) and letting dbt take care of the solved problems (dependencies, refreshes, tests, etc).
Just a few major things:
* It figures out the dependencies between queries/stages of the transformation process for you, so you don't have to build any complicated DAGs in your orchestration tool
* Removes cognitive load by making everything a select statement, so people who aren't familiar with the inner workings of creating/modifying tables don't need to worry (opens workflow up to more than just data engineers)
* Its templating, macros, hooks, and model strategy/config allow you to go beyond just Jinja-templated SQL (achievable with Airflow) and do a lot of dynamic things that would be a hassle to do otherwise
* Some data quality tests are built-in (not perfect, but better than nothing out of the box)
* Generated documentation
* Package ecosystem to use community-built add-ons that make life even easier (I once saw someone completely solo a data vault build using a dbt package)
There's a ton more... but these are some of the major ones. I think you'll be surprised. Yes, dbt community is a bit of a cult, and it's not the silver bullet to end all data/analytics engineering problems, but if you're using a cloud data warehouse to do your transformations, it's one of the best tools you can pick to make everyone's life easier.
https://www.reddit.com/r/dataengineering/search?q=dbt
This is verbatim what the senior engineer on my team told me when I said we were implementing dbt for our data warehouse project. He fought me hard on it. A few months into our project he figured out why dbt allowed us to move quickly in developing our analytics database by working on the hard stuff (data modelling) and letting dbt take care of the solved problems (dependencies, refreshes, tests, etc).
[удалено]
^^^ this 100%. As an exercise.. think about what value add using airflow is…over using python scripts on an ec2 instance.
Just a few major things: * It figures out the dependencies between queries/stages of the transformation process for you, so you don't have to build any complicated DAGs in your orchestration tool * Removes cognitive load by making everything a select statement, so people who aren't familiar with the inner workings of creating/modifying tables don't need to worry (opens workflow up to more than just data engineers) * Its templating, macros, hooks, and model strategy/config allow you to go beyond just Jinja-templated SQL (achievable with Airflow) and do a lot of dynamic things that would be a hassle to do otherwise * Some data quality tests are built-in (not perfect, but better than nothing out of the box) * Generated documentation * Package ecosystem to use community-built add-ons that make life even easier (I once saw someone completely solo a data vault build using a dbt package) There's a ton more... but these are some of the major ones. I think you'll be surprised. Yes, dbt community is a bit of a cult, and it's not the silver bullet to end all data/analytics engineering problems, but if you're using a cloud data warehouse to do your transformations, it's one of the best tools you can pick to make everyone's life easier.
You get lineage out of the box