Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Column Level Lineage: SELECT * EXCEPT() not showing all columns (BigQuery) #576

Open
phalcon22 opened this issue Feb 5, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@phalcon22
Copy link

Describe the bug

  • When using SELECT * EXCEPT(), the column level lineage is not detailed.

SQL

CREATE OR REPLACE TEMP TABLE `table_1` AS

SELECT
    * EXCEPT(B)
FROM `dataset.table_0`
;

To Reproduce
Note here we refer to SQL provided in prior step as stored in a file named test.sql

import json
from sqllineage.core.metadata.dummy import DummyMetaDataProvider
from sqllineage.runner import LineageRunner

with open("test.sql") as f:
    sp = f.read()

with open('metadata.json') as file:
    metadata = json.load(file)
provider = DummyMetaDataProvider(metadata)

lineage = LineageRunner(sp, dialect="bigquery", metadata_provider=provider, verbose=True)
lineage.print_column_lineage()

Metadata.json:

{
    "dataset.table_0": [
        "A",
        "B",
        "C",
        "D"
      ]
}

Result

<default>.table_1.* except(b) <- dataset.table_0.* except(b)

Expected behavior

<default>.table_1.A <- dataset.table_0.A
<default>.table_1.C <- dataset.table_0.C
<default>.table_1.D <- dataset.table_0.D

Python version (available via python --version)

  • 3.9.12

SQLLineage version (available via sqllineage --version):

  • 1.5.1
@phalcon22 phalcon22 added the bug Something isn't working label Feb 5, 2024
@reata
Copy link
Owner

reata commented Feb 6, 2024

Bug confirmed. * EXCEPT(B) is parsed as column name, which is clearly wrong. To fix the issue:

  1. we need to correctly recognize this is wildcard, so expand wildcard with metadata functionality works.
  2. handle except during wildcard expansion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants