Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Issue with on_schema_change config #273

Merged
merged 2 commits into from
Feb 4, 2025

Conversation

maladroitthief
Copy link
Contributor

@maladroitthief maladroitthief commented Jan 28, 2025

Summary

This fixes the on_schema_change config

Description

  • Moves the raw_on_schema_change variable back into scope for the config validator

Test Results

Reproduced the bug described in #268, made changes to a local version of dbt-dremio and confirmed that it was behaving as expected.

Changelog

  • Added a summary of what this PR accomplishes to CHANGELOG.md

Related Issue

resolves: #268

@looping-aba
Copy link

Hello,
We've faced the same issue and applied the variable instanciation earlier in the incremental.sql macro.
For future non regression test it may be positive to enrich dremio adapter tests (test_incremental.py) with the dbt-test-adapters basic test.

class TestIncrementalOnSchemaChange(BaseIncrementalOnSchemaChange):
    pass

Regards

Copy link
Contributor

@bcmeireles bcmeireles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the contribution! I allowed the pipeline to run and it looks good. Could you also add a change to CHANGELOG.md about this bug fix and add the test mentioned by @looping-aba please?

@maladroitthief
Copy link
Contributor Author

Thank you for the contribution! I allowed the pipeline to run and it looks good. Could you also add a change to CHANGELOG.md about this bug fix and add the test mentioned by @looping-aba please?

Changes have been made, I am in the process of setting up the test environment locally to make sure the newly added test case passes

@bcmeireles
Copy link
Contributor

Thank you for the contribution! I allowed the pipeline to run and it looks good. Could you also add a change to CHANGELOG.md about this bug fix and add the test mentioned by @looping-aba please?

Changes have been made, I am in the process of setting up the test environment locally to make sure the newly added test case passes

Just a small correction, the current version in development is 1.8.2 so these changes would still be under it and not 1.8.3 :)

bcmeireles
bcmeireles previously approved these changes Jan 29, 2025
Copy link
Contributor

@bcmeireles bcmeireles left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thank you for your contribution :)

@maladroitthief
Copy link
Contributor Author

@bcmeireles hang on, it looks like I have the test import wrong. Still working on the local setup, but it failed in CI

- Moves the `raw_on_schema_change` variable back into scope for the
  config validator
- Adds `BaseIncrementalOnSchemaChange` test to test_incremental.py

resolves: dremio#268
@maladroitthief
Copy link
Contributor Author

Success after running pytest tests/functional/adapter/basic

================================================================================ test session starts =================================================================================
platform linux -- Python 3.9.20, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/ian-weller/workspace/maladroitthief/dbt-dremio, configfile: pytest.ini
collected 28 items

tests/functional/adapter/basic/test_adapter_methods.py .                                                                                                                       [  3%]
tests/functional/adapter/basic/test_base_mat.py .                                                                                                                              [  7%]
tests/functional/adapter/basic/test_concurrency.py .                                                                                                                           [ 10%]
tests/functional/adapter/basic/test_current_timestamp.py ..                                                                                                                    [ 17%]
tests/functional/adapter/basic/test_data_types.py .                                                                                                                            [ 21%]
tests/functional/adapter/basic/test_dbt_show.py ....                                                                                                                           [ 35%]
tests/functional/adapter/basic/test_docs_generate.py ...                                                                                                                       [ 46%]
tests/functional/adapter/basic/test_empty.py .                                                                                                                                 [ 50%]
tests/functional/adapter/basic/test_ephemeral.py .                                                                                                                             [ 53%]
tests/functional/adapter/basic/test_generic_tests.py .                                                                                                                         [ 57%]
tests/functional/adapter/basic/test_incremental.py .......                                                                                                                     [ 82%]
tests/functional/adapter/basic/test_override.py .                                                                                                                              [ 85%]
tests/functional/adapter/basic/test_singular_ephemeral.py .                                                                                                                    [ 89%]
tests/functional/adapter/basic/test_singular_tests.py .                                                                                                                        [ 92%]
tests/functional/adapter/basic/test_snapshots.py ss                                                                                                                            [100%]

================================================================================== warnings summary ==================================================================================
../../../.pyenv/versions/3.9.20/envs/dbt-dremio/lib/python3.9/site-packages/_pytest/config/__init__.py:1296
  /home/ian-weller/.pyenv/versions/3.9.20/envs/dbt-dremio/lib/python3.9/site-packages/_pytest/config/__init__.py:1296: PytestConfigWarning: Unknown config option: env_files

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================================================ 26 passed, 2 skipped, 1 warning in 288.55s (0:04:48) ================================================================

@99Lys 99Lys self-requested a review January 30, 2025 15:05
Copy link
Contributor

@99Lys 99Lys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks nice! Thank you for your contribution!!

@howareyouman
Copy link
Contributor

@maladroitthief Hey!
Thank you for your contribution!
Before merging this PR, please sign Contributor License Agreement, enabling Dremio to distribute your contribution without restriction.
Thank you beforehand!

@maladroitthief
Copy link
Contributor Author

@howareyouman just signed it

@howareyouman howareyouman merged commit 050a948 into dremio:main Feb 4, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: on_schema_change takes no effect
5 participants