From 0a1d5548da14933611586e627ec6b40585a37fec Mon Sep 17 00:00:00 2001 From: Daniel Herrera Date: Wed, 17 Jul 2024 12:12:28 +0200 Subject: [PATCH] docs: Update quickstart guide for using cosmos in airflow workflow for dbt transformations branding and style --- ...ransformations-in-airflow-with-cosmos.adoc | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/modules/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc b/modules/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc index c89f8a603..4e84cb1c4 100644 --- a/modules/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc +++ b/modules/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc @@ -10,7 +10,7 @@ == Overview -This tutorial demonstrates how to install Apache Airflow on a local machine, configure the workflow to use dbt teradata to run dbt transformations using astronomer cosmos library, and run it against a Teradata Vantage database. Apache Airflow is a task scheduling tool that is typically used to build data pipelines to process and load data. https://astronomer.github.io/astronomer-cosmos/[Astronomer cosmos] library simplify transformations with dbt in Apache Airflow. Using Cosmos, run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code. +This tutorial demonstrates how to install Apache Airflow on a local machine, configure the workflow to use dbt teradata to run dbt transformations using the astronomer cosmos library, and run it against a Teradata Vantage database. Apache Airflow is a task scheduling tool that is typically used to build data pipelines to process and load data. https://astronomer.github.io/astronomer-cosmos/[Astronomer cosmos] library simplifies orchestrating dbt data transformations in Apache Airflow. Using Cosmos, allows running dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code. In this example, we will explain how to use astronomer cosmos to run dbt teradata transformations in airflow against Teradata vantage database. == Prerequisites @@ -70,7 +70,7 @@ pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL} -- ==== -3. Install the Apache Airflow Teradata provider stable version from PyPI repository. +3. Install the Apache Airflow Teradata provider stable version from the corresponding PyPI repository. + [source, bash] ---- @@ -155,13 +155,13 @@ cp -r jaffle_shop $AIRFLOW_HOME/dags/dbt/ [source, bash] export dbt_project_home_dir=../../jaffle_shop + -NOTE: Change `/../../` to path of jaffle_shop project path. +NOTE: You might need to change `/../../` to the specific path where you cloned the jaffle_shop project. -3. Define the virtual environment path where dbt-teradata installed in <> as an environment variable `dbt_venv_dir`. +3. Define the path to the virtual environment where dbt-teradata was installed as an environment variable `dbt_venv_dir`. [source, bash] set dbt_venv_dir=/../../dbt_env/bin/dbt + -NOTE: Change `/../../` to path where virtual environment path defined. +NOTE: You might need to change `/../../` to the specific path where the virtual environment is located. == Start Apache Airflow web server 1. Run airflow web server @@ -185,7 +185,8 @@ airflow standalone * Password (required): lake_user == Define DAG in Apache Airflow -Dags in airflow will be defined as python file. Similarly, define below dag to run dbt transformations defined in jaffle_shop dbt project against Teradata Vantage system using cosmos. Copy below python code and save it as `airflow-cosmos-dbt-teradata-integration.py` under directory $AIRFLOW_HOME/files/dags. +Dags in airflow are defined as python files. The dag below runs the dbt transformations defined in the `jaffle_shop` dbt project on a Teradata Vantage system using cosmos.Copy the python code below and save it as `airflow-cosmos-dbt-teradata-integration.py` under the directory $AIRFLOW_HOME/files/dags. + [source, python] ---- import os @@ -228,17 +229,17 @@ with DAG( == Load DAG -When the dag file is copied to $AIRFLOW_HOME/files/dags, apache airflow loads the dag to airflow UI. +When the dag file is copied to $AIRFLOW_HOME/files/dags, Apache Airflow loads the dag to airflow UI. == Run DAG -Run the dag as shown in below image. +Run the dag as shown in the image below. image::{dir}/airflow-dag.png[Run dag,align="left" width=75%] == Summary -In this quick start guide, we explored how to utilize Astronomer Cosmos library in apache airflow to execute dbt teradata transformations against Teradata Vantage instance. +In this quick start guide, we explored how to utilize Astronomer Cosmos library in Apache Airflow to execute dbt teradata transformations against a Teradata Vantage instance. == Further reading * link:https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html[Apache Airflow DAGs reference]