Skip to content

Latest commit

 

History

History
86 lines (44 loc) · 7.53 KB

quickstart.md

File metadata and controls

86 lines (44 loc) · 7.53 KB

Quickstart

Prerequisites:

  1. Github account (signup)
  2. Google account (for access to Big Query)
  3. Gitpod.io account (signup)

Steps

Setup GCP Big Query project and Github repo to open on Gitpod

  1. Create a Big Query (BQ) GCP Project and download a service account key credentials json file with BQ access permissions. If you already have an existing GCP BQ Project but not a service account or a key credentials json file, see these for instructions on creating one. Download and save this file in your local directory as you will need this later to setup your gitpod.io dbt workspace.

  2. (Optional) Install the Gitpod Chrome/Brave/Firefox browser extension/addon. This will add a Gitpod button on your main repository page that will conveniently open your github project in a gitpod.io workspace.

  3. If you already have an existing DBT project on github, open the github project in gitpod.io using the Gitpod button on your github page. Note: You can also create a new branch or use an existing branch if you don't want to use the default branch to open in gitpod.io -- just add the tree/<branch-name> to end of the gitpod url.

  4. If you don't have an existing dbt project but just want to try out using dbt on gitpod, create a new dbt project from an existing dbt project template repository. Then you can open this dbt project on gitpod.

  5. As an alternative to clicking the gitpod button (i.e. you didn't install the browser extension), you can also open your github project in a gitpod workspace by using the url https://gitpod.io/#<your-github-repo-url> (if you are not using the main branch, you can append the name of the branch of the repo (/tree/<branch-name>) to the end of the url).

Note: If your git repo is not hosted by Github, but is supported by gitpod (e.g. bitbucket and gitlab), please follow gitpod.io's instructions on how to load those repos into your gitpod workspace.

Configuring your dbt project environment in Gitpod.io

Terminal

  1. Run the following command in your gitpod VS Code terminal window. This script will copy over some configuration files (including a gitpod configuration file called .gitpod.yml ).
curl -s https://raw.githubusercontent.com/butchland/dbt-gitpod-config/main/scripts/loadconfig | bash
  1. Edit the gitpod-dbt-profiles.yml configuration files to match your dbt profile. Make sure it is using the correct profile name (line 1) as the profile property defined in your dbt_project.yml file. Also check that the dataset property is an existing BQ dataset in your GCP project. If you generated your dbt project from the sample dbt template linked above, they have been configured to match the default profile (sample_dbt_learn_project). Just make sure to create a jaffle_shop dataset in your GCP BQ project.

Gitpod dbt profiles and dbt project

  1. Create a Gitpod workspace environment variable named CREDENTIALS_JSON with the scope limited to the project (usually <user-name>/<repo-name>). Copy the contents of your service account key credentials json file (which you should have created in step 1 of the Github and GCP BQ setup section) into the value of the environment variable.

Environment Var

  1. Commit your new changes into git (i.e. git add, git commit ) and do a git push to update your remote git repository.

  2. At this point, your dbt project is now configured for use in a gitpod workspace. You can close and delete the gitpod workspace instance of your project (since all the changes are now in your dbt project repo).

Opening and testing your dbt project environment in your Gitpod workspace

VS Code Workspace

  1. From your remote git repository web page, click the Gitpod button (if you installed the gitpod browser extension) or alternatively, open the url https://gitpod.io/#<your-github-repo-url>.

  2. With .gitpod.yml in your project root directory, gitpod will now run the set of commands in the .gitpod.yml file to configure your environment for use with dbt, including the setup of a virtual python environment, your dbt profile, as well as other steps such as installing a default set of vscode extensions to make your gitpod vscode editor a powerful dbt development environment.

  3. The first run takes a little bit longer than the usual, as it takes additional steps to use it for the first time. See the gitpod documentation regarding these steps.

  4. If you have setup everything correctly (including reloading the window to pickup the installation of the vscode extensions), there should be a VS Code browser showing the generated dbt documentation about your project.

  5. You can setup the virtual environment venv/bin/python as default python interpreter to be used by project.

  6. Open a new terminal and check if everything is setup correctly by running dbt debug. If should return an OK Connection success message.

dbt debug message

  1. You should now be able to run other dbt commands just like in a local dbt cli environment.

Using the VS Code DBT Extensions

  1. Open a model or analysis sql file. There should be additional navigational menu buttons in the top right provided by the VS Code DBT Big Query Power User extension. You can trigger model compilation or run the sql as well as preview the results. Opening the query runner button runs the sql and previews the results of the sql. For more information, click here

VS Code DBT Biq Query Power User Extension

Lastly, several additional VS Code extensions that improve your dbt development experience have also been installed in your workspace environment:

  • vscode-dbt - VSCode Snippet Package for jinja and dbt
  • sqlfluff - A linter and auto-formatter for SQLfluff, a popular linting tool for SQL and dbt

Conclusion

At this point, you should have a pretty powerful and productive online development environment that can rival your local development environment.

If you have comments or suggestions to improve this documentation, please file an issue on the tracker.