Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify web and mobile model #3

Merged
merged 2 commits into from
Oct 24, 2023
Merged

Unify web and mobile model #3

merged 2 commits into from
Oct 24, 2023

Conversation

agnessnowplow
Copy link
Collaborator

@agnessnowplow agnessnowplow commented Sep 18, 2023

Status:
Integration tests are added (Redshift tested locally). Not expecting more changes apart from PR points.

optional modules:

  • originally not part of v1 but time allowed to transfer all of them, the web ones pass int tests, the app error did not have one and needed some rework so best to check that closely

Internal Documentation PR:
-snowplow/documentation#631 (comment)

Copy link
Contributor

@matus-tomlein matus-tomlein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good!

macros/bigquery/field_lists.sql Outdated Show resolved Hide resolved
macros/bigquery/field_lists.sql Outdated Show resolved Hide resolved
models/users/scratch/snowplow_unified_users_this_run.sql Outdated Show resolved Hide resolved
models/users/scratch/snowplow_unified_users_this_run.sql Outdated Show resolved Hide resolved
@agnessnowplow agnessnowplow marked this pull request as ready for review October 4, 2023 09:08
Copy link
Contributor

@matus-tomlein matus-tomlein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (at least from what I can review), amazing work!

macros/field_extractions/get_browser_context_fields.sql Outdated Show resolved Hide resolved
macros/field_extractions/get_browser_context_fields.sql Outdated Show resolved Hide resolved
macros/field_extractions/get_browser_context_fields.sql Outdated Show resolved Hide resolved
macros/field_extractions/get_browser_context_fields.sql Outdated Show resolved Hide resolved
macros/field_extractions/get_browser_context_fields.sql Outdated Show resolved Hide resolved
macros/bigquery/field_lists.sql Outdated Show resolved Hide resolved
macros/bigquery/field_lists.sql Outdated Show resolved Hide resolved
macros/bigquery/field_lists.sql Outdated Show resolved Hide resolved
macros/bigquery/field_lists.sql Outdated Show resolved Hide resolved
macros/bigquery/field_lists.sql Outdated Show resolved Hide resolved
Copy link
Contributor

@rlh1994 rlh1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we're tidying things, just as few ideas:

  • We don't really need the cluster_by macros anymore now the models are all a single file, we could just add them directly to the config?
  • I really don't get why allow_refresh is not part of the utils package at this point...
  • Once it's settled I'd like to just have a look at all our macros and work out if there's any we could combine, make more flexible, combine in file only (to make it clearer where things are maybe), or pull back into the models now that we don't have 4 versions of each file (where they don't differ by warehouse). Also just consistency things like we should make sure all columns don't have trailing commas etc.

macros/field_extractions/get_consent_fields.sql Outdated Show resolved Hide resolved
macros/config_check.sql Outdated Show resolved Hide resolved
@agnessnowplow
Copy link
Collaborator Author

While we're tidying things, just as few ideas:

  • We don't really need the cluster_by macros anymore now the models are all a single file, we could just add them directly to the config?
  • I really don't get why allow_refresh is not part of the utils package at this point...
  • Once it's settled I'd like to just have a look at all our macros and work out if there's any we could combine, make more flexible, combine in file only (to make it clearer where things are maybe), or pull back into the models now that we don't have 4 versions of each file (where they don't differ by warehouse). Also just consistency things like we should make sure all columns don't have trailing commas etc.
  • good idea, I have removed the cluster by macros.
  • it makes sense to add the allow_refresh to utils, however I would wait to do this in combination with other macro changes that we decide to do to avoid having to deal with package dependency issues and multiple minor upgrades
  • let's have a workshop with all AEs and go through our packages and decide on a plan to make the merges / cleanups

@agnessnowplow
Copy link
Collaborator Author

Update on the state: Thanks for all the feedback so far! Things should solidify from now as everything has been added including integration tests and even the optional modules from both web and mobile that was not even planned at this stage. The app errors module still needs revision as it does not come with integration tests and I merged the this run tables for it. Other than that I consider it done (minus a final revision on wording and local final redshift tests).

Copy link
Contributor

@rlh1994 rlh1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing in my review is actually blocking, pretty much entirely nitpicks to try and find a single thing wrong, this is amazing work

README.md Show resolved Hide resolved
dbt_project.yml Outdated Show resolved Hide resolved
integration_tests/.scripts/integration_test.sh Outdated Show resolved Hide resolved
docs/markdown/snowplow_unified_macros_docs.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
integration_tests/README.md Outdated Show resolved Hide resolved
.github/workflows/no-response.yml Outdated Show resolved Hide resolved
.github/pull_request_template.md Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
.github/pull_request_template.md Outdated Show resolved Hide resolved
.github/pull_request_template.md Show resolved Hide resolved
.github/pull_request_template.md Outdated Show resolved Hide resolved
Copy link
Contributor

@rlh1994 rlh1994 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@agnessnowplow agnessnowplow merged commit 980f21a into main Oct 24, 2023
@emielver emielver deleted the feature/unify branch December 1, 2023 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants