Skip to content

Commit

Permalink
docs: Update quickstart guide for using cosmos in airflow workflow fo…
Browse files Browse the repository at this point in the history
…r dbt transformations branding and style
  • Loading branch information
Daniel-Itzul committed Jul 17, 2024
1 parent 7ab6393 commit 0a1d554
Showing 1 changed file with 10 additions and 9 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

== Overview

This tutorial demonstrates how to install Apache Airflow on a local machine, configure the workflow to use dbt teradata to run dbt transformations using astronomer cosmos library, and run it against a Teradata Vantage database. Apache Airflow is a task scheduling tool that is typically used to build data pipelines to process and load data. https://astronomer.github.io/astronomer-cosmos/[Astronomer cosmos] library simplify transformations with dbt in Apache Airflow. Using Cosmos, run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code.
This tutorial demonstrates how to install Apache Airflow on a local machine, configure the workflow to use dbt teradata to run dbt transformations using the astronomer cosmos library, and run it against a Teradata Vantage database. Apache Airflow is a task scheduling tool that is typically used to build data pipelines to process and load data. https://astronomer.github.io/astronomer-cosmos/[Astronomer cosmos] library simplifies orchestrating dbt data transformations in Apache Airflow. Using Cosmos, allows running dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code.
In this example, we will explain how to use astronomer cosmos to run dbt teradata transformations in airflow against Teradata vantage database.

== Prerequisites
Expand Down Expand Up @@ -70,7 +70,7 @@ pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}
--
====

3. Install the Apache Airflow Teradata provider stable version from PyPI repository.
3. Install the Apache Airflow Teradata provider stable version from the corresponding PyPI repository.
+
[source, bash]
----
Expand Down Expand Up @@ -155,13 +155,13 @@ cp -r jaffle_shop $AIRFLOW_HOME/dags/dbt/
[source, bash]
export dbt_project_home_dir=../../jaffle_shop
+
NOTE: Change `/../../` to path of jaffle_shop project path.
NOTE: You might need to change `/../../` to the specific path where you cloned the jaffle_shop project.

3. Define the virtual environment path where dbt-teradata installed in <<Install dbt>> as an environment variable `dbt_venv_dir`.
3. Define the path to the virtual environment where dbt-teradata was installed as an environment variable `dbt_venv_dir`.
[source, bash]
set dbt_venv_dir=/../../dbt_env/bin/dbt
+
NOTE: Change `/../../` to path where virtual environment path defined.
NOTE: You might need to change `/../../` to the specific path where the virtual environment is located.

== Start Apache Airflow web server
1. Run airflow web server
Expand All @@ -185,7 +185,8 @@ airflow standalone
* Password (required): lake_user

== Define DAG in Apache Airflow
Dags in airflow will be defined as python file. Similarly, define below dag to run dbt transformations defined in jaffle_shop dbt project against Teradata Vantage system using cosmos. Copy below python code and save it as `airflow-cosmos-dbt-teradata-integration.py` under directory $AIRFLOW_HOME/files/dags.
Dags in airflow are defined as python files. The dag below runs the dbt transformations defined in the `jaffle_shop` dbt project on a Teradata Vantage system using cosmos.Copy the python code below and save it as `airflow-cosmos-dbt-teradata-integration.py` under the directory $AIRFLOW_HOME/files/dags.

[source, python]
----
import os
Expand Down Expand Up @@ -228,17 +229,17 @@ with DAG(

== Load DAG

When the dag file is copied to $AIRFLOW_HOME/files/dags, apache airflow loads the dag to airflow UI.
When the dag file is copied to $AIRFLOW_HOME/files/dags, Apache Airflow loads the dag to airflow UI.

== Run DAG

Run the dag as shown in below image.
Run the dag as shown in the image below.

image::{dir}/airflow-dag.png[Run dag,align="left" width=75%]

== Summary

In this quick start guide, we explored how to utilize Astronomer Cosmos library in apache airflow to execute dbt teradata transformations against Teradata Vantage instance.
In this quick start guide, we explored how to utilize Astronomer Cosmos library in Apache Airflow to execute dbt teradata transformations against a Teradata Vantage instance.

== Further reading
* link:https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html[Apache Airflow DAGs reference]
Expand Down

0 comments on commit 0a1d554

Please sign in to comment.