Skip to content

Commit

Permalink
Merge pull request #92 from JustMaris/patch-1
Browse files Browse the repository at this point in the history
Updating unique_table_sync_key to address incremental unique test fail
  • Loading branch information
fivetran-joemarkiewicz authored Oct 17, 2023
2 parents 526c2d9 + edc146c commit 6864ac3
Showing 12 changed files with 33 additions and 22 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@

env/
target/
dbt_modules/
logs/
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,13 @@
# dbt_fivetran_log v1.2.0

[PR #92](https://github.com/fivetran/dbt_fivetran_log/pull/92) includes the following updates:
## Bug Fixes
- The `unique_table_sync_key` surrogate key which is created within the `fivetran_platform__audit_table` has been updated to also be comprised of the `schema_name` in addition to the `connector_id`, `destination_id`, `table_name`, `write_to_table_start` fields. This update will also ensure the uniqueness test on this record is accurately testing the true grain of the model.
- 🚨 Please be aware that as the `fivetran_platform__audit_table` model is an incremental model a `--full-refresh` will be needed following the package upgrade in order for this change to properly be applied to all records in the end model. 🚨

## Contributors
- [@JustMaris](https://github.com/JustMaris) ([#92](https://github.com/fivetran/dbt_fivetran_log/pull/92))

# dbt_fivetran_log v1.1.0

[PR #87](https://github.com/fivetran/dbt_fivetran_log/pull/87) includes the following updates:
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -66,7 +66,7 @@ Include the following Fivetran Platform package version range in your `packages.
```yaml
packages:
- package: fivetran/fivetran_log
version: [">=1.1.0", "<1.2.0"]
version: [">=1.2.0", "<1.3.0"]
```

> Note that altough the source connector is now "Fivetran Platform", the package retains the old name of "fivetran_log".
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
config-version: 2
name: 'fivetran_log'
version: '1.1.0'
version: '1.2.0'
require-dbt-version: [">=1.3.0", "<2.0.0"]

models:
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions docs/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/run_results.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'fivetran_log_integration_tests'
version: '1.1.0'
version: '1.2.0'

config-version: 2
profile: 'integration_tests'
1 change: 1 addition & 0 deletions integration_tests/requirements.txt
Original file line number Diff line number Diff line change
@@ -6,3 +6,4 @@ dbt-spark>=1.3.0,<2.0.0
dbt-spark[PyHive]>=1.3.0,<2.0.0
dbt-databricks>=1.3.0,<2.0.0

oscrypto @ git+https://github.com/wbond/oscrypto.git@d5f3437
2 changes: 1 addition & 1 deletion models/fivetran_platform.yml
Original file line number Diff line number Diff line change
@@ -272,7 +272,7 @@ models:
table completely, as `fivetran_audit` reported on pre-duplicated data loads.
columns:
- name: unique_table_sync_key
description: Primary key of this table, hashed on `connector_id`, `destination_id`, `table_name`, and `unique_table_sync_key`.
description: Primary key of this table, hashed on `schema_name`, `connector_id`, `destination_id`, `table_name`, and `unique_table_sync_key`.
tests:
- unique
- not_null
4 changes: 2 additions & 2 deletions models/fivetran_platform__audit_table.sql
Original file line number Diff line number Diff line change
@@ -134,10 +134,10 @@ final as (

select
*,
{{ dbt_utils.generate_surrogate_key(['connector_id', 'destination_id', 'table_name', 'write_to_table_start']) }} as unique_table_sync_key, -- for incremental materialization
{{ dbt_utils.generate_surrogate_key(['schema_name','connector_id', 'destination_id', 'table_name', 'write_to_table_start']) }} as unique_table_sync_key, -- for incremental materialization
{{ dbt.date_trunc('day', 'sync_start') }} as sync_start_day -- for partitioning in databricks
from sum_records_modified
)

select *
from final
from final

0 comments on commit 6864ac3

Please sign in to comment.