Skip to content

Commit

Permalink
Unify web and mobile model
Browse files Browse the repository at this point in the history
  • Loading branch information
agnessnowplow committed Sep 21, 2023
1 parent 73eba98 commit 9df2c51
Show file tree
Hide file tree
Showing 173 changed files with 20,976 additions and 1 deletion.
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @agnessnowplow
68 changes: 68 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
---
name: Bug report
about: Report a bug or an issue you've found with this package
title: ''
labels: type:bug
assignees: ''

---

## Describe the bug
<!---
A clear and concise description of what the bug is.
--->

## Steps to reproduce
<!---
In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example model code, etc is all very helpful here.
--->

## Expected results
<!---
A clear and concise description of what you expected to happen.
--->

## Actual results
<!---
A clear and concise description of what actually happens.
--->

## Screenshots and log output
<!---
If applicable, add screenshots or log output to help explain your problem.
--->

## System information
**The contents of your `packages.yml` file:**
```yml
# contents goes here
```

**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [ ] bigquery
- [ ] snowflake
- [ ] databricks
- [ ] other (specify: ____________)

**The output of `dbt --version`:**
```
<output goes here>
```

**The operating system you're using:**


**The output of `python --version`:**


## Additional context
<!---
Add any other context about the problem here. For example, if you think you know which line of code is causing the issue.
--->

## Are you interested in contributing towards the fix?
<!---
Let us know if you want to contribute towards the fix, and whether you would need a hand getting started
--->
25 changes: 25 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
---
name: Feature request
about: Suggest an idea for this package
title: ''
labels: type:enhancement
assignees: ''

---

## Is your feature request related to a problem? Please describe.
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

## Describe the solution you'd like
<!-- A clear and concise description of what you want to happen. -->

## Describe alternatives you've considered
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

## Additional context
<!-- Add any other context or screenshots about the feature request here. -->

## Are you interested in contributing towards this feature?
<!---
Let us know if you want to contribute towards the feature, and whether you would need a hand getting started
--->
16 changes: 16 additions & 0 deletions .github/ISSUE_TEMPLATE/question.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
---
name: Question
about: A more general question about the package. Consider using discourse if more applicable.
title: ''
labels: type:question
assignees: ''

---

<!-- Please consider if your question would be better placed on the Snowplow Discourse (https://discourse.snowplow.io/) for more open conversations -->

### Question
<!-- What is your question -->

### Additional context
<!-- Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here. -->
81 changes: 81 additions & 0 deletions .github/labels.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
- name: "category:breaking_change"
color: '1e1037'
description: "A breaking change will be introduced if this issue is completed."
- name: "category:documentation"
color: '3849b8'
description: "About the documentation."
- name: "cla:no"
color: '313131'
description: "[Auto generated] Snowplow Contributor License Agreement has not been signed."
- name: "cla:yes"
color: 'c1c1c1'
description: "[Auto generated] Snowplow Contributor License Agreement has been signed."
- name: "good first issue"
color: '69f6ff'
description: "Good issue for a first time contributor."
- name: "priority:high"
color: 'f1ff1d'
description: "To fix as soon as possible."
- name: "priority:low"
color: 'ffd0d4'
description: "Not on the roadmap."
- name: "priority:medium"
color: 'ff6372'
description: "On the roadmap."
- name: "status:blocked"
color: '042e31'
description: "Maintainers can't work on this right now."
- name: "status:completed"
color: '0ea417'
description: "Completed - but might not be released yet."
- name: "status:duplicate"
color: 'b6e1e3'
description: "Duplicates another issue."
- name: "status:has_pr"
color: '0ea462'
description: "A PR exists for this issue."
- name: "status:in_progress"
color: '0e9ba4'
description: "Maintainers are working on this."
- name: "status:needs_triage"
color: 'b6e3cf'
description: "Needs maintainer triage."
- name: "status:wont_fix"
color: '04311d'
description: "Can't or won't be fixed."
- name: "status:info_needed"
color: '56B9BF'
description: "More information is needed from filer for issue to be actionable"
- name: "status:do_not_merge"
color: 'D93F0B'
description: "Flag to denote a Issue or PR which should not yet be merged (usually pending a release)"
- name: "status:help_wanted"
color: '69f6ff'
description: "Help is wanted to help solve this issue."
- name: "type:admin"
color: 'fdffd2'
description: 'About code or repo management.'
- name: "type:bug"
color: '4c381f'
description: 'Bugs or weaknesses. The issue has to contain steps to reproduce.'
- name: "type:enhancement"
color: 'ffbd69'
description: 'New features or improvements to existing features.'
- name: "type:question"
color: 'ffebd2'
description: "Questions to the maintainers."
- name: "category:models"
color: '6738b8'
description: "Related to the models in the package."
- name: "category:macros"
color: 'f1ff1d'
description: "Related to the macros in the package."
- name: "category:tests"
color: 'a738b8'
description: "Related to the tests in the package."
- name: "category:other"
color: '6372ff'
description: "Related to other parts of the package or repo itself."
#- name: "category:4"
# color: '9e62dd'
# description: "Desc"
22 changes: 22 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
<!--
If this is your first time contributing you will be asked to sign the Individual Contributor License Agreement.
If you would prefer to read this in advance of submitting your PR you can find it here https://docs.google.com/forms/d/e/1FAIpQLSd89YTDQ1XpTZbj3LpOkquV_h1Y8k9ay3iFbJsZsJrz18I23Q/viewform
-->

## Description & motivation
<!--
Describe your changes, and why you're making them.
-->

## Checklist
- [ ] I have verified that these changes work locally
- [ ] I have updated the README.md (if applicable)
- [ ] I have added tests & descriptions to my models (and macros if applicable)
- [ ] I have raised a [documentation](https://github.com/snowplow/documentation) PR if applicable (Link here if required)
- [ ] Is your change a breaking change?

<!--
## Release Only Checklist
- [ ] I have updated the version number in all relevant places
- [ ] I have updated the CHANGELOG.md
-->
26 changes: 26 additions & 0 deletions .github/workflows/no-response.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: No Response

# Both `issue_comment` and `scheduled` event types are required for this Action
# to work properly.
on:
issue_comment:
types: [created]
schedule:
# Schedule for 08:00 every day
- cron: '0 8 * * *'

jobs:
noResponse:
runs-on: ubuntu-latest
steps:
- uses: lee-dohm/no-response@v0.5.0
with:
token: ${{ github.token }}
daysUntilClose: 14
responseRequiredLabel: "status:info_needed"
closeComment: >
This issue has been automatically closed because there has been no response
to our request for more information from the original author within 14 days. With only what
is currently in the issue, we don't have enough information
to take action. Please comment with this information if you have it to reopen the issue so
that we can investigate further.
138 changes: 138 additions & 0 deletions .github/workflows/pr_tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
name: pr_tests

on:
pull_request:

concurrency: dbt_integration_tests

env:
# Set profiles.yml directory
DBT_PROFILES_DIR: ./ci

# Redshift Connection
REDSHIFT_TEST_HOST: ${{ secrets.REDSHIFT_TEST_HOST }}
REDSHIFT_TEST_USER: ${{ secrets.REDSHIFT_TEST_USER }}
REDSHIFT_TEST_PASS: ${{ secrets.REDSHIFT_TEST_PASS }}
REDSHIFT_TEST_DBNAME: ${{ secrets.REDSHIFT_TEST_DBNAME }}
REDSHIFT_TEST_PORT: ${{ secrets.REDSHIFT_TEST_PORT }}

# BigQuery Connection
BIGQUERY_TEST_DATABASE: ${{ secrets.BIGQUERY_TEST_DATABASE }}
BIGQUERY_LOCATION: ${{ secrets.BIGQUERY_LOCATION }}
BIGQUERY_SERVICE_TYPE: ${{ secrets.BIGQUERY_SERVICE_TYPE }}
BIGQUERY_SERVICE_PROJECT_ID: ${{ secrets.BIGQUERY_SERVICE_PROJECT_ID }}
BIGQUERY_SERVICE_PRIVATE_KEY_ID: ${{ secrets.BIGQUERY_SERVICE_PRIVATE_KEY_ID }}
BIGQUERY_SERVICE_PRIVATE_KEY: ${{ secrets.BIGQUERY_SERVICE_PRIVATE_KEY }}
BIGQUERY_SERVICE_CLIENT_EMAIL: ${{ secrets.BIGQUERY_SERVICE_CLIENT_EMAIL }}
BIGQUERY_SERVICE_CLIENT_ID: ${{ secrets.BIGQUERY_SERVICE_CLIENT_ID }}
BIGQUERY_SERVICE_AUTH_URI: ${{ secrets.BIGQUERY_SERVICE_AUTH_URI }}
BIGQUERY_SERVICE_TOKEN_URI: ${{ secrets.BIGQUERY_SERVICE_TOKEN_URI }}
BIGQUERY_SERVICE_AUTH_PROVIDER_X509_CERT_URL: ${{ secrets.BIGQUERY_SERVICE_AUTH_PROVIDER_X509_CERT_URL }}
BIGQUERY_SERVICE_CLIENT_X509_CERT_URL: ${{ secrets.BIGQUERY_SERVICE_CLIENT_X509_CERT_URL }}

# Snowflake Connection
SNOWFLAKE_TEST_ACCOUNT: ${{ secrets.SNOWFLAKE_TEST_ACCOUNT }}
SNOWFLAKE_TEST_USER: ${{ secrets.SNOWFLAKE_TEST_USER }}
SNOWFLAKE_TEST_PASSWORD: ${{ secrets.SNOWFLAKE_TEST_PASSWORD }}
SNOWFLAKE_TEST_ROLE: ${{ secrets.SNOWFLAKE_TEST_ROLE }}
SNOWFLAKE_TEST_DATABASE: ${{ secrets.SNOWFLAKE_TEST_DATABASE }}
SNOWFLAKE_TEST_WAREHOUSE: ${{ secrets.SNOWFLAKE_TEST_WAREHOUSE }}

# Postgres Connection
POSTGRES_TEST_HOST: ${{ secrets.POSTGRES_TEST_HOST }}
POSTGRES_TEST_USER: ${{ secrets.POSTGRES_TEST_USER }}
POSTGRES_TEST_PASS: ${{ secrets.POSTGRES_TEST_PASS }}
POSTGRES_TEST_PORT: ${{ secrets.POSTGRES_TEST_PORT }}
POSTGRES_TEST_DBNAME: ${{ secrets.POSTGRES_TEST_DBNAME }}

# Databricks Connection
DATABRICKS_TEST_HOST: ${{ secrets.DATABRICKS_TEST_HOST }}
DATABRICKS_TEST_HTTP_PATH: ${{ secrets.DATABRICKS_TEST_HTTP_PATH }}
DATABRICKS_TEST_TOKEN: ${{ secrets.DATABRICKS_TEST_TOKEN }}
DATABRICKS_TEST_ENDPOINT: ${{ secrets.DATABRICKS_TEST_ENDPOINT }}

jobs:
pr_tests:
name: pr_tests
runs-on: ubuntu-latest
defaults:
run:
# Run tests from integration_tests sub dir
working-directory: ./integration_tests
strategy:
matrix:
dbt_version: ["1.*"]
warehouse: ["postgres", "bigquery", "snowflake", "databricks"] # TODO: Add RS self-hosted runner

services:
postgres:
image: postgres:latest
env:
POSTGRES_DB: ${{ secrets.POSTGRES_TEST_DBNAME }}
POSTGRES_USER: ${{ secrets.POSTGRES_TEST_USER }}
POSTGRES_PASSWORD: ${{ secrets.POSTGRES_TEST_PASS }}
# Set health checks to wait until postgres has started
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
# Maps tcp port 5432 on service container to the host
- 5432:5432

steps:
- name: Check out
uses: actions/checkout@v3

# Remove '*' and replace '.' with '_' in DBT_VERSION & set as SCHEMA_SUFFIX.
# SCHEMA_SUFFIX allows us to run multiple versions of dbt in parallel without overwriting the output tables
- name: Set SCHEMA_SUFFIX env
run: echo "SCHEMA_SUFFIX=$(echo ${DBT_VERSION%.*} | tr . _)" >> $GITHUB_ENV
env:
DBT_VERSION: ${{ matrix.dbt_version }}

- name: Set DEFAULT_TARGET env
run: |
echo "DEFAULT_TARGET=${{ matrix.warehouse }}" >> $GITHUB_ENV
- name: Python setup
uses: actions/setup-python@v4
with:
python-version: "3.8.x"

- name: Pip cache
uses: actions/cache@v3
with:
path: ~/.cache/pip
key: ${{ runner.os }}-pip-${{ matrix.dbt_version }}-${{ matrix.warehouse }}
restore-keys: |
${{ runner.os }}-pip-${{ matrix.dbt_version }}-${{ matrix.warehouse }}
# Install latest patch version. Upgrade if cache contains old patch version.
- name: Install dependencies
run: |
pip install --upgrade pip wheel setuptools
pip install -Iv dbt-${{ matrix.warehouse }}==${{ matrix.dbt_version }} --upgrade
dbt deps
if: ${{matrix.warehouse != 'spark'}}


- name: Install spark dependencies
run: |
pip install --upgrade pip wheel setuptools
pip install -Iv "dbt-${{ matrix.warehouse }}[ODBC]"==${{ matrix.dbt_version }} --upgrade
dbt deps
if: ${{matrix.warehouse == 'spark'}}

- name: "Pre-test: Drop ci schemas"
run: |
dbt run-operation post_ci_cleanup --target ${{ matrix.warehouse }}
- name: Run tests
run: ./.scripts/integration_test.sh -d ${{ matrix.warehouse }}

# post_ci_cleanup sits in utils package
- name: "Post-test: Drop ci schemas"
run: |
dbt run-operation post_ci_cleanup --target ${{ matrix.warehouse }}
Loading

0 comments on commit 9df2c51

Please sign in to comment.