Fix iceberg full refresh#247 #252

sanga8 · 2023-10-30T14:42:38Z

Resolves #247

Description

In the incremental.sql file, it was first checking if the relation was iceberg, instead of checking for full refresh.
See athena connector implementation for reference.

We fix it by modifying the order of the condition, we first check if the user has used the full refresh flag (i.e. -f or --full-refresh).

To check if the user used the flag, we use the dbt core macro should_full_refresh() instead of the currently implemented full_refresh_mode variable. Since we don't need the full_refresh_mode variable anymore, we remove this variable from the code.

Once the full refresh condition is True, we check if it is an iceberg materialization. If yes, we first delete the table using glue__drop_relation macro in adapters.sql. Then, we write the table with the iceberg_write function, that will re-create the table from scratch since it does not exist anymore.

TLDR:

We modify the order of condition by first checking if full refresh is enabled instead of if iceberg.
We check if full refresh is enabled with a dbt-core macro instead of custom implementation.
If full refresh and iceberg, then we delete the table.
We re-write the table using iceberg write.

Checklist

I have signed the CLA
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
I have updated the CHANGELOG.md and added information about my change to the "dbt-glue next" section.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Fix single_functional test

Bumps [dbt-tests-adapter](https://github.com/dbt-labs/dbt-core) from 1.6.6 to 1.6.7. - [Release notes](https://github.com/dbt-labs/dbt-core/releases) - [Changelog](https://github.com/dbt-labs/dbt-core/blob/v1.6.7/CHANGELOG.md) - [Commits](dbt-labs/dbt-core@v1.6.6...v1.6.7) --- updated-dependencies: - dependency-name: dbt-tests-adapter dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

…t-tests-adapter-1.6.7 Bump dbt-tests-adapter from 1.6.6 to 1.6.7

Bumps [dbt-tests-adapter](https://github.com/dbt-labs/dbt-core) from 1.6.7 to 1.7.0. - [Release notes](https://github.com/dbt-labs/dbt-core/releases) - [Changelog](https://github.com/dbt-labs/dbt-core/blob/v1.7.0/CHANGELOG.md) - [Commits](dbt-labs/dbt-core@v1.6.7...v1.7.0) --- updated-dependencies: - dependency-name: dbt-tests-adapter dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

…l_test.py to avoid basename conflict

…t-tests-adapter-1.7.0 Bump dbt-tests-adapter from 1.6.7 to 1.7.0

Fixed tests

Bumps [dbt-tests-adapter](https://github.com/dbt-labs/dbt-core) from 1.7.0 to 1.7.1. - [Release notes](https://github.com/dbt-labs/dbt-core/releases) - [Changelog](https://github.com/dbt-labs/dbt-core/blob/v1.7.1/CHANGELOG.md) - [Commits](dbt-labs/dbt-core@v1.7.0...v1.7.1) --- updated-dependencies: - dependency-name: dbt-tests-adapter dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com>

Bumps [black](https://github.com/psf/black) from 23.10.1 to 23.11.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](psf/black@23.10.1...23.11.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>

[test] Refine unit test and integ tests

[doc] Remove unnecessary extra_py_files for Delta Lake

[minor][doc] Fix Iceberg config in README

* fix test generic * Add test on seed in generic tests

* fix test generic * Add test on seed in generic tests * Roll back table model

dbt/include/glue/macros/materializations/incremental/incremental.sql

…l/adapter (aws-samples#275) * Migrated test_single_functional.py from test_docs.py, and from test_basic_functional_test.py to test_snapshot.py * Separated database names * Separated database names

Bumps [mypy](https://github.com/python/mypy) from 1.6.1 to 1.7.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v1.6.1...v1.7.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Updates the requirements on [moto](https://github.com/getmoto/moto) to permit the latest version. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@4.2.7...4.2.8) --- updated-dependencies: - dependency-name: moto dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Reorganizing commits * Added TestSnapshotTimestampGlue * Added TestDocsGenReferencesGlue * fix TestTableMatGlue and TestValidateConnectionGlue * Skip TestBaseCachingGlue Also this fixes the execution error --------- Co-authored-by: menuetb <83284881+menuetb@users.noreply.github.com> Co-authored-by: Akira Ajisaka <akiraaj@amazon.com>

)

* Fix integ test for GitHub actions * Separated database in test_docs.py * Separated database in test_snapshot.py * Run integ tests for a database with random name * Removed unneeded test * Moving unique_schema, profiles_config_update, and cleanup to conftest.py * More refactor * No need to update proifile_config. Remove * Now that we are using different schema for each class, need to cleanup after the test * Update test_docs.py and test_snapshot.py as well * Create separate conftest.py for integration tests * Use helper method as possible --------- Co-authored-by: Akira Ajisaka <akiraaj@amazon.com>

…ga8/dbt-glue into fix-iceberg-full-refresh#247

Vincent.PAUWELS and others added 30 commits October 30, 2023 15:22

first commit

4e26166

revert pr template

caeca5e

fix when existing_relation_type is none

80fa1d7

Fixed unit tests test_glue_session.py and test_glue_session_demo.py

fa85d32

Added debug logs and fixed some tests

199d622

Fix single_functional test

036955b

Merge pull request aws-samples#1 from aajisaka/fix_unit_tests

78106a4

Fix single_functional test

Fixed functional_test

c095ac6

Merge pull request aws-samples#256 from aws-samples/dependabot/pip/db…

6a51320

…t-tests-adapter-1.6.7 Bump dbt-tests-adapter from 1.6.6 to 1.6.7

Fixed functional/adapter/test_basic.py

bb2ffe3

Fixed exception handling and cursor issue

df43bc1

Renamed functional_test/adapter/test_basic.py to test_basic_functiona…

cefed9e

…l_test.py to avoid basename conflict

Updated README

398333d

Merge pull request aws-samples#257 from aws-samples/dependabot/pip/db…

07b6100

…t-tests-adapter-1.7.0 Bump dbt-tests-adapter from 1.6.7 to 1.7.0

Merge pull request aws-samples#253 from moomindani/fix_unit_tests

03f2d24

Fixed tests

Update to version 1.7

4f2a5e5

Added CI

f028c83

now correctly delete the table

7f401c7

cleaner

7295153

Update CI

60713bd

Update CI

9b34097

Update CI

fd06f6b

Update CI

4985298

Update CI

fe8d07a

Update CI

5c587b3

Update CI

5b0ba6f

menuetb and others added 10 commits November 10, 2023 11:33

Merge pull request aws-samples#268 from moomindani/refine_tests

d095ec8

[test] Refine unit test and integ tests

[minor][doc] Fix Iceberg config in README

203892f

Merge pull request aws-samples#267 from aajisaka/fix-delta-py-file

f5c40fc

[doc] Remove unnecessary extra_py_files for Delta Lake

Merge pull request aws-samples#269 from aajisaka/fix-iceberg-doc

d98dc06

[minor][doc] Fix Iceberg config in README

fix test generic (aws-samples#270)

8c663b6

fix test generic (aws-samples#271)

bdc6070

Fix test generic (aws-samples#272)

975167f

* fix test generic * Add test on seed in generic tests

beauty

25b6bfe

duplicate line of code

4d12ac1

Fix test generic (aws-samples#276)

3c054d0

* fix test generic * Add test on seed in generic tests * Roll back table model

nicor88 reviewed Nov 13, 2023

View reviewed changes

dbt/include/glue/macros/materializations/incremental/incremental.sql Outdated Show resolved Hide resolved

moomindani and others added 16 commits November 13, 2023 12:30

Migrate snapshot test and docs test from functional_test to functiona…

ef5cdf0

…l/adapter (aws-samples#275) * Migrated test_single_functional.py from test_docs.py, and from test_basic_functional_test.py to test_snapshot.py * Separated database names * Separated database names

use drop_relation instead of glue__drop_relation

7c9c412

Revert the change in get_result_from_cursor to fix test (aws-samples#279

879f41a

)

first commit

97ca609

revert pr template

4249e08

fix when existing_relation_type is none

20949b7

now correctly delete the table

bef16ed

cleaner

0e4e7c1

beauty

a5cd470

duplicate line of code

3e43a6e

use drop_relation instead of glue__drop_relation

b611bb5

Merge branch 'fix-iceberg-full-refresh#247' of https://github.com/san…

9083b21

…ga8/dbt-glue into fix-iceberg-full-refresh#247

menuetb added the enable-functional-tests This label enable functional tests label Nov 22, 2023

menuetb approved these changes Dec 14, 2023

View reviewed changes

menuetb merged commit fbd53bb into aws-samples:main Dec 14, 2023
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix iceberg full refresh#247 #252

Fix iceberg full refresh#247 #252

sanga8 commented Oct 30, 2023 •

edited

Loading

Fix iceberg full refresh#247 #252

Fix iceberg full refresh#247 #252

Conversation

sanga8 commented Oct 30, 2023 • edited Loading

Description

Checklist

sanga8 commented Oct 30, 2023 •

edited

Loading