Added quick start guide to use cosmos in airflow workflow for dbt transformations. #211
Conversation
|
| * Python 3.8, 3.9, 3.10 or 3.11 installed. | ||
| * python3-env, python3-pip, pipx installed. | ||
| [source, bash] | ||
| sudo apt install -y python3-venv python3-pip pipx |
There was a problem hiding this comment.
I think this command is different for each OS. Should we split it according to that?
There was a problem hiding this comment.
Addressed the comment.
| + | ||
| [source, bash] | ||
| ---- | ||
| export AIRFLOW_HOME=~/airflow |
There was a problem hiding this comment.
Should we differentiate the commands between installing airflow locally vs. in a virtual environment?
There was a problem hiding this comment.
Totally agree with @sunilkmallam let's install it only on venv, we don't want the user to tamper their default python install.
Let's create venv first.
There was a problem hiding this comment.
Included installation on venv.
| [source, bash] | ||
| export dbt_project_home_dir=../../jaffle_shop | ||
| + | ||
| NOTE: Change `/../../` to path of jaffle_shop project path. |
There was a problem hiding this comment.
Can we change "path of jaffle_shop project path" to path of "your jaffle_shop project path".
There was a problem hiding this comment.
NOTE is not required as per latest change. Removed the Note.
|
|
||
| == Summary | ||
|
|
||
| In this quick start guide, we explored how to utilize Astronomer Cosmos library in apache airflow to execute dbt teradata transformations against Teradata Vantage instance. |
There was a problem hiding this comment.
dbt teradata transformations should be replaced with dbt transformations
…r dbt transformations branding and style
| + | ||
| include::ROOT:partial$vantage_clearscape_analytics.adoc[] | ||
| * Python 3.8, 3.9, 3.10 or 3.11 installed. | ||
| * python3-env, python3-pip, pipx installed. |
There was a problem hiding this comment.
| * python3-env, python3-pip, pipx installed. | |
| * python3-env, python3-pip, pipx installed. |
This is only applicable for linux mac comves with env and pip, we are never using pipx in the guide, not sure why we are installing it.
There was a problem hiding this comment.
Added instructions for Mac, Windows and Linux as tabs
| * Python 3.8, 3.9, 3.10 or 3.11 installed. | ||
| * python3-env, python3-pip, pipx installed. | ||
| [source, bash] | ||
| sudo apt install -y python3-venv python3-pip pipx |
There was a problem hiding this comment.
Same as above this is only applicable to linux. Mac doesn't need it and Windows is not supported, only through WSL which is linux anyway.
There was a problem hiding this comment.
Added note for WSL on Windows. Wherever tab is required for three platforms, added tabs and removed tabs at not required places.
| + | ||
| [source, bash] | ||
| ---- | ||
| export AIRFLOW_HOME=~/airflow |
There was a problem hiding this comment.
Totally agree with @sunilkmallam let's install it only on venv, we don't want the user to tamper their default python install.
Let's create venv first.
| ---- | ||
| 2. Install `apache-airflow` stable version 2.9.2 from PyPI repository.: | ||
| + | ||
| [tabs] |
There was a problem hiding this comment.
All of the blocks are the same because mac is as linux a Unix type OS and windows is on WSL which is not linux as well, same as in the other guide not sure we need to repeat on the three tabs. Maybe we just make a comment that windows is WSL so windows on WSL is the same as linux, and that mac is also the same, the only difference would be installing pip and env which don't come as default in python for linux.
There was a problem hiding this comment.
Addressed the comment
| ---- | ||
|
|
||
| == Install dbt | ||
| 1. Create a new python environment to manage dbt and its dependencies. Activate the environment: |
There was a problem hiding this comment.
This should be at the beginning.
There was a problem hiding this comment.
@JH255095 do want to place Install dbt section at the beginning? Currently placed after Install Airflow step.
| == Install dbt | ||
| 1. Create a new python environment to manage dbt and its dependencies. Activate the environment: | ||
| + | ||
| [tabs] |
There was a problem hiding this comment.
Again same as above, we might not need the tabs since all is the same.
| airflow standalone | ||
| ---- | ||
| 2. Access the airflow UI. Visit https://localhost:8080 in the browser and log in with the admin account details shown in the terminal. | ||
|
|
There was a problem hiding this comment.
Add image of the terminal
| ---- | ||
| 2. Access the airflow UI. Visit https://localhost:8080 in the browser and log in with the admin account details shown in the terminal. | ||
|
|
||
| == Define Apache Airflow connection to Vantage Cloud Lake |
There was a problem hiding this comment.
This one let's make it more generic, assume that you are working with CSAE, this is applicable to lake as it appears in the compute clusters guide, but here we don't have all the section regarding the SQL needed for creating the user, if we just point that you need a teradata environment and you can get one on CSAE we save all of that part. We just need to modify this section then to use the CSAE host, user and password.
There was a problem hiding this comment.
Modified as more generic.
...es/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc
Outdated
Show resolved
Hide resolved
…mations-in-airflow-with-cosmos.adoc
...es/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc
Outdated
Show resolved
Hide resolved
…mations-in-airflow-with-cosmos.adoc
...es/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc
Outdated
Show resolved
Hide resolved
…mations-in-airflow-with-cosmos.adoc
...es/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc
Outdated
Show resolved
Hide resolved
…mations-in-airflow-with-cosmos.adoc
...es/other-integrations/pages/execute-dbt-teradata-transformations-in-airflow-with-cosmos.adoc
Outdated
Show resolved
Hide resolved
…mations-in-airflow-with-cosmos.adoc
Added quick start guide to use cosmos in airflow workflow for dbt transformations.