Fix iceberg full refresh#247 (#252)

* first commit * revert pr template * fix when existing_relation_type is none * Fixed unit tests test_glue_session.py and test_glue_session_demo.py * Added debug logs and fixed some tests * Fix single_functional test * Bump dbt-tests-adapter from 1.6.6 to 1.6.7 Bumps [dbt-tests-adapter](https://github.com/dbt-labs/dbt-core) from 1.6.6 to 1.6.7. - [Release notes](https://github.com/dbt-labs/dbt-core/releases) - [Changelog](https://github.com/dbt-labs/dbt-core/blob/v1.6.7/CHANGELOG.md) - [Commits](dbt-labs/dbt-core@v1.6.6...v1.6.7) --- updated-dependencies: - dependency-name: dbt-tests-adapter dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Fixed functional_test * Bump dbt-tests-adapter from 1.6.7 to 1.7.0 Bumps [dbt-tests-adapter](https://github.com/dbt-labs/dbt-core) from 1.6.7 to 1.7.0. - [Release notes](https://github.com/dbt-labs/dbt-core/releases) - [Changelog](https://github.com/dbt-labs/dbt-core/blob/v1.7.0/CHANGELOG.md) - [Commits](dbt-labs/dbt-core@v1.6.7...v1.7.0) --- updated-dependencies: - dependency-name: dbt-tests-adapter dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fixed functional/adapter/test_basic.py * Fixed exception handling and cursor issue * Renamed functional_test/adapter/test_basic.py to test_basic_functional_test.py to avoid basename conflict * Updated README * Update to version 1.7 * Added CI * Bump dbt-tests-adapter from 1.7.0 to 1.7.1 Bumps [dbt-tests-adapter](https://github.com/dbt-labs/dbt-core) from 1.7.0 to 1.7.1. - [Release notes](https://github.com/dbt-labs/dbt-core/releases) - [Changelog](https://github.com/dbt-labs/dbt-core/blob/v1.7.1/CHANGELOG.md) - [Commits](dbt-labs/dbt-core@v1.7.0...v1.7.1) --- updated-dependencies: - dependency-name: dbt-tests-adapter dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Bump black from 23.10.1 to 23.11.0 Bumps [black](https://github.com/psf/black) from 23.10.1 to 23.11.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](psf/black@23.10.1...23.11.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * now correctly delete the table * cleaner * Update CI * Update CI * Update CI * Update CI * Update CI * Update CI * Update CI * Update CI * Increased timeout for integ test * Update unit tests and CI * Added unit test for Lake Formation * Added test for Glue default version * Renamed test class * Updated dependency for 1.7.1 * Configured concurrency control for tests * Configured concurrency control for tests * Configured concurrency control for tests * Change requirements * fix spark version * Update tox.ini * Add integration test for all PR * Add labeled based functional test * Remove unnecessary extra_py_files for Delta Lake * Added unit test for test_hudi_merge_table with Mock using moto and mock_glue * Moved S3Url util class to util.py * Renamed util.py for unit tests * Update CHANGELOG * Fix unit test * Fixed test_generic_tests * [minor][doc] Fix Iceberg config in README * fix test generic (#270) * fix test generic (#271) * Fix test generic (#272) * fix test generic * Add test on seed in generic tests * beauty * duplicate line of code * Fix test generic (#276) * fix test generic * Add test on seed in generic tests * Roll back table model * Migrate snapshot test and docs test from functional_test to functional/adapter (#275) * Migrated test_single_functional.py from test_docs.py, and from test_basic_functional_test.py to test_snapshot.py * Separated database names * Separated database names * use drop_relation instead of glue__drop_relation * Bump mypy from 1.6.1 to 1.7.0 (#273) Bumps [mypy](https://github.com/python/mypy) from 1.6.1 to 1.7.0. - [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md) - [Commits](python/mypy@v1.6.1...v1.7.0) --- updated-dependencies: - dependency-name: mypy dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update moto requirement from ~=4.2.7 to ~=4.2.8 (#277) Updates the requirements on [moto](https://github.com/getmoto/moto) to permit the latest version. - [Release notes](https://github.com/getmoto/moto/releases) - [Changelog](https://github.com/getmoto/moto/blob/master/CHANGELOG.md) - [Commits](getmoto/moto@4.2.7...4.2.8) --- updated-dependencies: - dependency-name: moto dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * all basic tests enabled (#278) * Reorganizing commits * Added TestSnapshotTimestampGlue * Added TestDocsGenReferencesGlue * fix TestTableMatGlue and TestValidateConnectionGlue * Skip TestBaseCachingGlue Also this fixes the execution error --------- Co-authored-by: menuetb <83284881+menuetb@users.noreply.github.com> Co-authored-by: Akira Ajisaka <akiraaj@amazon.com> * Revert the change in get_result_from_cursor to fix test (#279) * Fix integ tests (#280) * Fix integ test for GitHub actions * Separated database in test_docs.py * Separated database in test_snapshot.py * Run integ tests for a database with random name * Removed unneeded test * Moving unique_schema, profiles_config_update, and cleanup to conftest.py * More refactor * No need to update proifile_config. Remove * Now that we are using different schema for each class, need to cleanup after the test * Update test_docs.py and test_snapshot.py as well * Create separate conftest.py for integration tests * Use helper method as possible --------- Co-authored-by: Akira Ajisaka <akiraaj@amazon.com> * first commit * revert pr template * fix when existing_relation_type is none * now correctly delete the table * cleaner * beauty * duplicate line of code * use drop_relation instead of glue__drop_relation --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Vincent.PAUWELS <vincent.pauwels@airliquide.com> Co-authored-by: Noritaka Sekiyama <sekiyama@amazon.com> Co-authored-by: Akira Ajisaka <akiraaj@amazon.com> Co-authored-by: Noritaka Sekiyama <moomindani@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: menuetb <83284881+menuetb@users.noreply.github.com> Co-authored-by: Anders <anders.swanson@dbtlabs.com>
aws-samples · Dec 14, 2023 · fbd53bb · fbd53bb
1 parent 0031d57
commit fbd53bb
Show file tree

Hide file tree

Showing 2 changed files with 14 additions and 12 deletions.
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
@@ -21,4 +21,4 @@ resolves #
 - [ ] I have updated the `CHANGELOG.md` and added information about my change to the "dbt-glue next" section.
 
 
-By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
+By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
diff --git a/dbt/include/glue/macros/materializations/incremental/incremental.sql b/dbt/include/glue/macros/materializations/incremental/incremental.sql
@@ -20,10 +20,6 @@
   {%- set expire_snapshots = config.get('iceberg_expire_snapshots', 'True') -%}
   {%- set table_properties = config.get('table_properties', default='empty') -%}
 
-
-  {%- set full_refresh_config = config.get('full_refresh', default=False) -%}
-  {%- set full_refresh_mode = (flags.FULL_REFRESH == 'True' or full_refresh_config == 'True') -%}
-
   {% set target_relation = this %}
   {% set existing_relation_type = adapter.get_table_type(target_relation)  %}
   {% set tmp_relation = make_temp_relation(target_relation, '_tmp') %}
@@ -41,12 +37,6 @@
         {%- set hudi_options = config.get('hudi_options', default={}) -%}
         {{ adapter.hudi_merge_table(target_relation, sql, unique_key, partition_by, custom_location, hudi_options, substitute_variables) }}
         {% set build_sql = "select * from " + target_relation.schema + "." + target_relation.identifier + " limit 1 "%}
-  {% elif file_format == 'iceberg' %}
-        {{ adapter.iceberg_write(target_relation, sql, unique_key, partition_by, custom_location, strategy, table_properties) }}
-        {% set build_sql = "select * from glue_catalog." + target_relation.schema + "." + target_relation.identifier + " limit 1 "%}
-        {%- if expire_snapshots == 'True' -%}
-  	      {%- set result = adapter.iceberg_expire_snapshots(target_relation) -%}
-        {%- endif -%}
   {% else %}
       {% if strategy == 'insert_overwrite' and partition_by %}
         {% call statement() %}
@@ -57,17 +47,29 @@
         {% if file_format == 'delta' %}
             {{ adapter.delta_create_table(target_relation, sql, unique_key, partition_by, custom_location) }}
             {% set build_sql = "select * from " + target_relation.schema + "." + target_relation.identifier + " limit 1 " %}
+        {% elif file_format == 'iceberg' %}
+            {{ adapter.iceberg_write(target_relation, sql, unique_key, partition_by, custom_location, strategy, table_properties) }}
+            {% set build_sql = "select * from glue_catalog." + target_relation.schema + "." + target_relation.identifier + " limit 1 "%}
         {% else %}
             {% set build_sql = create_table_as(False, target_relation, sql) %}
         {% endif %}
-      {% elif existing_relation_type == 'view' or full_refresh_mode %}
+      {% elif existing_relation_type == 'view' or should_full_refresh() %}
         {{ drop_relation(target_relation) }}
         {% if file_format == 'delta' %}
             {{ adapter.delta_create_table(target_relation, sql, unique_key, partition_by, custom_location) }}
             {% set build_sql = "select * from " + target_relation.schema + "." + target_relation.identifier + " limit 1 " %}
+        {% elif file_format == 'iceberg' %}
+            {{ adapter.iceberg_write(target_relation, sql, unique_key, partition_by, custom_location, strategy, table_properties) }}
+            {% set build_sql = "select * from glue_catalog." + target_relation.schema + "." + target_relation.identifier + " limit 1 "%}
         {% else %}
             {% set build_sql = create_table_as(False, target_relation, sql) %}
         {% endif %}
+      {% elif file_format == 'iceberg' %}
+        {{ adapter.iceberg_write(target_relation, sql, unique_key, partition_by, custom_location, strategy, table_properties) }}
+        {% set build_sql = "select * from glue_catalog." + target_relation.schema + "." + target_relation.identifier + " limit 1 "%}
+        {%- if expire_snapshots == 'True' -%}
+  	      {%- set result = adapter.iceberg_expire_snapshots(target_relation) -%}
+        {%- endif -%}
       {% else %}
         {{ glue__create_tmp_table_as(tmp_relation, sql) }}
         {% set is_incremental = 'True' %}