Skip to content

Commit 6b65f44

Browse files
howareyoumanravjotbrar
howareyouman
and
ravjotbrar
authored
Dbt 1.8 migration (#244)
### Summary dbt-core migration to 1.8.* version ### Description Summarisation of 2 PRs to migrate to dbt-core 1.8.* version ### Test Results I'm expecting it as a baseline for future improvements. Tests statistics after migration to dbt 1.8 (with CE Dremio on-prem version): tests: - caching: passing - component: 2 out of 4 are passing (these tests rely on both DC and Software dremio versions) - functional: - basic: passing with ignored snapshot tests. - dbt_clone: failing - dremio_specific: passing - empty: failing - grants: failing - query_comment: passing - relation: 1 out of 2 are passing - unit_testing: failing - utils: passing - hooks: 7 out of 12 are passing - simple_copy: passing - unit: passing ### Changelog - [x] migration to dbt-core 1.8 is finished - [x] some tests are failing still --------- Co-authored-by: Ravjot Brar <83892020+ravjotbrar@users.noreply.github.com>
1 parent 424cdd0 commit 6b65f44

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+414
-138
lines changed

‎CHANGELOG.md

+9-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,12 @@
1-
# dbt-dremio MAIN
1+
# dbt-dremio v1.8.0
2+
3+
## Changes
4+
5+
## Dependency
6+
7+
- [#222](https://github.com/dremio/dbt-dremio/issues/222) Upgrade dbt-core to 1.8.8 and dbt-tests-adapter to 1.8.0
8+
9+
## Features
210

311
- [#223](https://github.com/dremio/dbt-dremio/issues/224) Implement merge strategy for incremental materializations
412
- [#229](https://github.com/dremio/dbt-dremio/issues/229) Add max operator to get_relation_last_modified macro

‎README.md

+4-4
Original file line numberDiff line numberDiff line change
@@ -6,20 +6,20 @@ dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-agg
66

77
[Dremio docs for our dbt adapter ](https://github.com/dremio/dbt-dremio/wiki/Using-Materializations-with-Dremio)
88

9-
## dbt-dremio version 1.7.0
10-
119
The `dbt-dremio` package contains all of the code enabling dbt to work with [Dremio](https://www.dremio.com/). For more information on using dbt with Dremio, consult [the docs](https://docs.getdbt.com/reference/warehouse-profiles/dremio-profile).
1210

1311
The dbt-dremio package supports both Dremio Cloud and Dremio Software (versions 22.0 and later).
1412

15-
Version 1.7.0 of the dbt-dremio adapter is compatible with dbt-core versions 1.2.0 to 1.7.*.
13+
## dbt-dremio version 1.8.0
14+
15+
Version 1.8.0 of the dbt-dremio adapter is compatible with dbt-core versions 1.8.*.
1616

1717
> Prior to version 1.1.0b, dbt-dremio was created and maintained by [Fabrice Etanchaud](https://github.com/fabrice-etanchaud) on [their GitHub repo](https://github.com/fabrice-etanchaud/dbt-dremio). Code for using Dremio REST APIs was originally authored by [Ryan Murray](https://github.com/rymurr). Contributors in this repo are credited for laying the groundwork and maintaining the adapter till version 1.0.6.5. The dbt-dremio adapter is maintained and distributed by Dremio starting with version 1.1.0b.
1818
1919
## Getting started
2020

2121
- [Install dbt-dremio](https://docs.getdbt.com/reference/warehouse-setups/dremio-setup)
22-
- Version 1.7.0 of dbt-dremio requires dbt-core >= 1.2.0 and <= 1.7.*. Installing dbt-dremio will automatically upgrade existing dbt-core versions earlier than 1.2.0 to 1.7.*, or install dbt-core v1.7.0 if no version of dbt-core is found.
22+
- Version 1.8.0 of dbt-dremio requires dbt-core >= 1.8.*.
2323
- Read the [introduction](https://docs.getdbt.com/docs/introduction/) and [viewpoint](https://docs.getdbt.com/docs/about/viewpoint/)
2424

2525
## Join the dbt Community

‎THIRD_PARTY_LICENSES.txt

+4-3
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,10 @@
1313
charset-normalizer 3.1.0 MIT License
1414
click 8.1.3 BSD License
1515
colorama 0.4.6 BSD License
16-
dbt-core 1.7.0 Apache Software License
17-
dbt-dremio 1.7.0 Apache Software License
16+
dbt-core 1.8.8 Apache Software License
17+
dbt-dremio 1.8.0 Apache Software License
1818
dbt-extractor 0.4.1 Apache Software License
19-
dbt-tests-adapter 1.7.0 Apache Software License
19+
dbt-tests-adapter 1.8.0 Apache Software License
2020
exceptiongroup 1.1.1 MIT License
2121
future 0.18.3 MIT License
2222
grpclib 0.4.3 BSD License
@@ -42,6 +42,7 @@
4242
pyrsistent 0.19.3 MIT License
4343
pytest 7.2.2 MIT License
4444
python-dateutil 2.8.2 Apache Software License; BSD License
45+
python-dotenv 1.0.1 BSD License
4546
python-slugify 8.0.1 MIT License
4647
pytimeparse 1.1.8 MIT License
4748
pytz 2022.7.1 MIT License

‎dbt/adapters/dremio/__version__.py

+1-2
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,4 @@
99
# See the License for the specific language governing permissions and
1010
# limitations under the License.
1111

12-
version = "1.7.0"
13-
12+
version = "1.8.0"

‎dbt/adapters/dremio/api/authentication.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
from typing import Optional
1818
from abc import abstractmethod
1919

20-
from dbt.events import AdapterLogger
20+
from dbt.adapters.events.logging import AdapterLogger
2121

2222
logger = AdapterLogger("dremio")
2323

‎dbt/adapters/dremio/api/cursor.py

+10-2
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@
2525
)
2626
from dbt.adapters.dremio.api.parameters import Parameters
2727

28-
from dbt.events import AdapterLogger
28+
from dbt.adapters.events.logging import AdapterLogger
2929

3030
logger = AdapterLogger("dremio")
3131

@@ -39,11 +39,20 @@ def __init__(self, api_parameters: Parameters):
3939
self._rowcount = -1
4040
self._job_results = None
4141
self._table_results: agate.Table = None
42+
self._description = None
4243

4344
@property
4445
def parameters(self):
4546
return self._parameters
4647

48+
@property
49+
def description(self):
50+
return self._description
51+
52+
@description.setter
53+
def description(self, value):
54+
self._description = value
55+
4756
@property
4857
def closed(self):
4958
return self._closed
@@ -126,7 +135,6 @@ def _populate_rowcount(self):
126135

127136
while True:
128137
time.sleep(0.2)
129-
130138
if job_status_state != last_job_state:
131139
logger.debug(f"Job State = {job_status_state}")
132140

‎dbt/adapters/dremio/api/handle.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616
from dbt.adapters.dremio.api.parameters import Parameters
1717
from dbt.adapters.dremio.api.rest.endpoints import login
1818

19-
from dbt.events import AdapterLogger
19+
from dbt.adapters.events.logging import AdapterLogger
2020

2121
logger = AdapterLogger("dremio")
2222

‎dbt/adapters/dremio/api/rest/endpoints.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@
3535
from dbt.adapters.dremio.api.parameters import Parameters
3636
from dbt.adapters.dremio.api.rest.url_builder import UrlBuilder
3737

38-
from dbt.events import AdapterLogger
38+
from dbt.adapters.events.logging import AdapterLogger
3939

4040
logger = AdapterLogger("dremio")
4141

‎dbt/adapters/dremio/connections.py

+12-10
Original file line numberDiff line numberDiff line change
@@ -20,12 +20,14 @@
2020
from dbt.adapters.dremio.api.handle import DremioHandle
2121
from dbt.adapters.dremio.api.parameters import ParametersBuilder
2222

23+
from dbt_common.clients import agate_helper
24+
2325
import time
2426
import json
2527

26-
import dbt.exceptions
28+
import dbt_common.exceptions
2729
from dbt.adapters.sql import SQLConnectionManager
28-
from dbt.contracts.connection import AdapterResponse
30+
from dbt.adapters.contracts.connection import AdapterResponse
2931

3032
from dbt.adapters.dremio.api.rest.endpoints import (
3133
delete_catalog,
@@ -43,7 +45,7 @@
4345
DremioBadRequestException,
4446
)
4547

46-
from dbt.events import AdapterLogger
48+
from dbt.adapters.events.logging import AdapterLogger
4749

4850
logger = AdapterLogger("dremio")
4951

@@ -61,13 +63,13 @@ def exception_handler(self, sql):
6163
except Exception as e:
6264
logger.debug(f"Error running SQL: {sql}")
6365
self.release()
64-
if isinstance(e, dbt.exceptions.DbtRuntimeError):
66+
if isinstance(e, dbt_common.exceptions.DbtRuntimeError):
6567
# during a sql query, an internal to dbt exception was raised.
6668
# this sounds a lot like a signal handler and probably has
6769
# useful information, so raise it without modification.
6870
raise
6971

70-
raise dbt.exceptions.DbtRuntimeError(e)
72+
raise dbt_common.exceptions.DbtRuntimeError(e)
7173

7274
@classmethod
7375
def open(cls, connection):
@@ -184,7 +186,7 @@ def execute(
184186
if fetch:
185187
table = cursor.table
186188
else:
187-
table = dbt.clients.agate_helper.empty_table()
189+
table = agate_helper.empty_table()
188190

189191
return response, table
190192

@@ -254,10 +256,10 @@ def _create_folders(self, database, schema, api_parameters):
254256
except DremioAlreadyExistsException:
255257
logger.debug(f"Folder {folder} already exists.")
256258
except DremioBadRequestException as e:
257-
if "Can not create a folder inside a [SOURCE]" in e.message:
258-
logger.debug(f"Ignoring {e}")
259-
else:
260-
raise e
259+
if "Can not create a folder inside a [SOURCE]" in e.message:
260+
logger.debug(f"Ignoring {e}")
261+
else:
262+
raise e
261263

262264
def _create_path_list(self, database, schema):
263265
path = [database]

‎dbt/adapters/dremio/credentials.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
from dbt.adapters.base import Credentials
15+
from dbt.adapters.contracts.connection import Credentials
1616
from dataclasses import dataclass
1717
from typing import Optional
1818
from dbt.adapters.dremio.relation import DremioRelation

‎dbt/adapters/dremio/impl.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@
2929
Capability,
3030
)
3131
from dbt.adapters.sql.impl import DROP_RELATION_MACRO_NAME
32-
from dbt.events import AdapterLogger
32+
from dbt.adapters.events.logging import AdapterLogger
3333

3434
logger = AdapterLogger("dremio")
3535

‎dbt/include/dremio/macros/adapters/snapshot.sql

+5-1
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ limitations under the License.*/
2626

2727
when not matched
2828
then insert ({{ insert_cols_csv }})
29-
values ({{ insert_cols_csv }})
29+
values
30+
({% for column_name in insert_cols -%}
31+
DBT_INTERNAL_SOURCE.{{ column_name }}
32+
{%- if not loop.last %}, {%- endif %}
33+
{%- endfor %})
3034

3135
{% endmacro %}

‎dbt/include/dremio/macros/adapters/validate_sql.sql

+14
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
/*Copyright (C) 2022 Dremio Corporation
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.*/
14+
115
{% macro default__validate_sql(sql) -%}
216
{% call statement('validate_sql') -%}
317
explain plan for {{ sql }}

‎dbt/include/dremio/macros/materializations/incremental/strategies.sql

+4-14
Original file line numberDiff line numberDiff line change
@@ -60,23 +60,13 @@ limitations under the License.*/
6060
{%- endfor %}
6161
{% endif %}
6262

63-
64-
{%- set dest_col_names = dest_columns | map(attribute="name") -%}
65-
{%- set dest_cols_prefixed = [] -%}
66-
67-
{% for dc in dest_col_names -%}
68-
{% set this_key_match %}
69-
DBT_INTERNAL_SOURCE."{{ dc }}"
70-
{% endset %}
71-
{% do dest_cols_prefixed.append(this_key_match) %}
72-
{%- endfor %}
73-
74-
7563
when not matched then insert
7664
({{ dest_cols_csv }})
7765
values
78-
({{ dest_cols_prefixed | join(',') }})
79-
66+
({% for column_name in dest_columns | map(attribute="name") -%}
67+
DBT_INTERNAL_SOURCE.{{ column_name }}
68+
{%- if not loop.last %}, {%- endif %}
69+
{%- endfor %})
8070

8171
{% endmacro %}
8272

‎dbt/include/dremio/macros/materializations/seed/seed.sql

+1-4
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,7 @@ limitations under the License.*/
1717
{%- set identifier = model['alias'] -%}
1818
{%- set format = config.get('format', validator=validation.any[basestring]) or 'iceberg' -%}
1919
{%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}
20-
{%- set target_relation = api.Relation.create(identifier=identifier,
21-
schema=schema,
22-
database=database,
23-
type='table') -%}
20+
{%- set target_relation = this.incorporate(type='table')-%}
2421
{% set grant_config = config.get('grants') %}
2522

2623
{{ run_hooks(pre_hooks) }}

‎dbt/include/dremio/macros/materializations/table/table.sql

+1-4
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,7 @@ limitations under the License.*/
1717
{%- set identifier = model['alias'] -%}
1818
{%- set format = config.get('format', validator=validation.any[basestring]) or 'iceberg' -%}
1919
{%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}
20-
{%- set target_relation = api.Relation.create(identifier=identifier,
21-
schema=schema,
22-
database=database,
23-
type='table') -%}
20+
{%- set target_relation = this.incorporate(type='table') -%}
2421
{% set grant_config = config.get('grants') %}
2522
{{ run_hooks(pre_hooks) }}
2623

‎dbt/include/dremio/macros/materializations/view/create_or_replace_view.sql

-38
This file was deleted.

‎dbt/include/dremio/macros/materializations/view/view.sql

+1-2
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,7 @@ limitations under the License.*/
1919

2020
{%- set exists_as_view = (old_relation is not none and old_relation.is_view) -%}
2121

22-
{%- set target_relation = api.Relation.create(
23-
identifier=identifier, schema=schema, database=database, type='view') -%}
22+
{%- set target_relation = this.incorporate(type='view')-%}
2423

2524
{% set grant_config = config.get('grants') %}
2625

‎dbt/include/dremio/macros/utils/data_types.sql

+14
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,17 @@
1+
/*Copyright (C) 2022 Dremio Corporation
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.*/
14+
115
{% macro dremio__type_string() %}
216
{{ return(api.Column.translate_type("VARCHAR")) }}
317
{% endmacro %}

‎dev_requirements.txt

+3-1
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,9 @@ Babel==2.12.1
33
betterproto==1.2.5
44
certifi==2023.7.22
55
charset-normalizer==3.1.0
6-
dbt-tests-adapter==1.7.0
6+
dbt-core==1.8.8
7+
dbt-tests-adapter==1.8.0
8+
python-dotenv==1.0.1
79
exceptiongroup==1.1.1
810
future==0.18.3
911
grpclib==0.4.3

‎setup.py

+3-2
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@
2222

2323
package_name = "dbt-dremio"
2424

25-
package_version = "1.7.0"
25+
package_version = "1.8.0"
2626

2727
description = """The Dremio adapter plugin for dbt"""
2828

@@ -37,7 +37,8 @@
3737
packages=find_namespace_packages(include=["dbt", "dbt.*"]),
3838
include_package_data=True,
3939
install_requires=[
40-
"dbt-core>=1.2, <=1.7.13",
40+
"dbt-core>=1.8",
41+
"dbt-adapters>=1.0.0, <2.0.0",
4142
"requests>=2.31.0",
4243
],
4344
classifiers=[

0 commit comments

Comments
 (0)