Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iceberg reading with explicit Schema support #6124

Open
devinrsmith opened this issue Sep 25, 2024 · 3 comments
Open

Iceberg reading with explicit Schema support #6124

devinrsmith opened this issue Sep 25, 2024 · 3 comments
Assignees
Labels
core Core development tasks feature request New feature or request iceberg
Milestone

Comments

@devinrsmith
Copy link
Member

I believe we need to offer Iceberg reading support based on a user-specified Schema (likely a Schema that was sourced from some history in the Table, or some subset); in the context where a user is passing in String column renames, the keys of that map are tied to a specific Schema (in this way also, it can internally be converted into a field-id rename, which can be used across column renaming). It may also be important in an enterprise for them to establish and record the specific Schema a table was initially ingested as (for example, they may want to enforce that db.t("MyNamespace", "MyTable") produces the same exact output today as it did yesterday). It is not enough to simply record the latest, or even a specific, Snapshot because the schema of the Table may change without a new Snapshot - for example, when a column is renamed, the Table's Schema will be updated, but no new Snapshots will be created.

As a point of convention, it probably makes sense to assume these defaults (in order):

  1. If a schema is provided, use that schema
  2. If a snapshot is provided, use schema table.schemas().get(snapshot.schemaId())
  3. Otherwise, use schema table.schema(); note, this is not equivalent to table.schemas().get(table.currentSnapshot().schemaId()) for reasons mentioned above.
@devinrsmith devinrsmith added this to the Triage milestone Sep 25, 2024
@devinrsmith
Copy link
Member Author

I can also make an argument that our default should be:

  1. If a schema is provided, use that schema
  2. Otherwise, use schema table.schema()

IE, if providing a specific snapshot (and not a schema), the results would be the snapshot's data projected into the latest Table's schema.

This has the advantage that tableAdapter.read() would semantically be the same as tableAdapter.read(table.currentSnapshot()).

Regardless, in either regime, the user can specify the schema, and so achieve the behavior they desire.

@devinrsmith
Copy link
Member Author

The table.schema() regime by default also allows the current constructors from io.deephaven.iceberg.layout.IcebergBaseLayout (and derived) to still be valid; while we should be okay breaking these, we eventually need to get into state where we aren't breaking them.

@rcaudy rcaudy added core Core development tasks and removed triage labels Sep 25, 2024
@rcaudy rcaudy modified the milestones: Triage, 0.38.0 Sep 25, 2024
@devinrsmith
Copy link
Member Author

Q: does PartitionSpec come into play, and if yes, is it always the case that the schema is equal to PartitionSpec#schema?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core development tasks feature request New feature or request iceberg
Projects
None yet
Development

No branches or pull requests

3 participants