You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I believe we need to offer Iceberg reading support based on a user-specified Schema (likely a Schema that was sourced from some history in the Table, or some subset); in the context where a user is passing in String column renames, the keys of that map are tied to a specific Schema (in this way also, it can internally be converted into a field-id rename, which can be used across column renaming). It may also be important in an enterprise for them to establish and record the specific Schema a table was initially ingested as (for example, they may want to enforce that db.t("MyNamespace", "MyTable") produces the same exact output today as it did yesterday). It is not enough to simply record the latest, or even a specific, Snapshot because the schema of the Table may change without a new Snapshot - for example, when a column is renamed, the Table's Schema will be updated, but no new Snapshots will be created.
As a point of convention, it probably makes sense to assume these defaults (in order):
If a schema is provided, use that schema
If a snapshot is provided, use schema table.schemas().get(snapshot.schemaId())
Otherwise, use schema table.schema(); note, this is not equivalent to table.schemas().get(table.currentSnapshot().schemaId()) for reasons mentioned above.
The text was updated successfully, but these errors were encountered:
The table.schema() regime by default also allows the current constructors from io.deephaven.iceberg.layout.IcebergBaseLayout (and derived) to still be valid; while we should be okay breaking these, we eventually need to get into state where we aren't breaking them.
I believe we need to offer Iceberg reading support based on a user-specified Schema (likely a Schema that was sourced from some history in the Table, or some subset); in the context where a user is passing in String column renames, the keys of that map are tied to a specific Schema (in this way also, it can internally be converted into a field-id rename, which can be used across column renaming). It may also be important in an enterprise for them to establish and record the specific Schema a table was initially ingested as (for example, they may want to enforce that
db.t("MyNamespace", "MyTable")
produces the same exact output today as it did yesterday). It is not enough to simply record the latest, or even a specific, Snapshot because the schema of the Table may change without a new Snapshot - for example, when a column is renamed, the Table's Schema will be updated, but no new Snapshots will be created.As a point of convention, it probably makes sense to assume these defaults (in order):
table.schemas().get(snapshot.schemaId())
table.schema()
; note, this is not equivalent totable.schemas().get(table.currentSnapshot().schemaId())
for reasons mentioned above.The text was updated successfully, but these errors were encountered: