Dynamically Generating Airflow DAGS

Problem Statement: Consider we have 3 pipelines and each pipeline have 4 stages in our Data Engineering project and we want create Airflow DAGS for each and every stage for all the pipelines. We end-up with 12 DAGS. However that’s only for development environment. If we want DAGS for the staging and production environment as […]
Source: Imaginea

Leave a Reply

Your email address will not be published.


*