-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update begin delete #24528
base: master
Are you sure you want to change the base?
Update begin delete #24528
Conversation
|
Has passed presto-Iceberg delete tests In the current state |
2e9f2cd
to
a010a6b
Compare
a010a6b
to
9730465
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! (docs)
Pull branch, local doc build, looks good. Thanks!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The changes on the Iceberg side look good to me
presto-kudu/src/main/java/com/facebook/presto/kudu/KuduMetadata.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/plan/TableWriterNode.java
Outdated
Show resolved
Hide resolved
Can you update the PR description with a release note, something like:
|
b88fc93
9730465
to
b88fc93
Compare
b88fc93
to
c2aa039
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left a couple minor notes on comments to clean up.
For the "Release Notes", I think you only really need the "SPI Changes" section and I'm not sure you need to mention DeleteTableHandle as the usage seems internal instead of part of the Connector SPI. You could instead list a second bullet on the ConnectorHandleResolver change that was also needed.
presto-main/src/main/java/com/facebook/presto/execution/scheduler/TableWriteInfo.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/scheduler/TableWriteInfo.java
Outdated
Show resolved
Hide resolved
Kudu tests are failing. I expect you need to update the KuduHandleResolver similar to Iceberg. |
c2aa039
to
51099b3
Compare
Changes to update finishDelete to new types Fixed Iceberg Delete as proof of concept. Fixed missing serialization changes. Kudu fix and checkstyle recombine iceberg tablehandle and deletetablehandle, fix serialization Documentation and add delete pagesink Fix implementation of delete pagesink delete extra comments/changes update KuduTableHandle remove comments and unused handle add kudu resolver for delete handle
51099b3
to
399bba2
Compare
Description
Changes to separate ConnectorTableHandles for deletes into it's own interface (ConnectorDeleteTableHandle) similar to ConnectorInsertTableHandle
If you have a connector that is broken by these changes, please refer to the fixes to presto-iceberg as an example. If you do not implement beginDelete, the changes will not affect you.
Motivation and Context
Please refer to #24520
Impact
Metadata, ConnectorMetadata, MetadataManager changed
->downstream affects execution: TableWriteInfo, ExecutionWriterTarget, TableWriterNode
Create 2 new types: ConnectorDeleteTableHandle, DeleteTableHandle
Types Changed: TableWriterNode.DeleteHandle
Also affects finishDelete, getDeleteRowIdColumnHandle types (AbstractMockMetadata, DelegatingMetadataManager)
Other classes affected:
ConnectorPageSinkProvider -> provide new pagesink for deletes?
ConnectorHandleResolver -> provide new getDeleteTableHandleClass method -> HandleResolver implements new ConnectorDeleteTableHandle
All connectors affected, those that implement DELETE will require more changes.
Kudu and iceberg connector both need to change types used for begin/finish delete
Iceberg currently uses IcebergTableHandle which inherits from basehivetablehandle->connectorTableHandle, and we would need to create a separate IcebergTableHandle for deletes This would eventually apply to update as well if were going to make the code symmetric
Test Plan
Pass presto-icebergs tests (includes DELETE tests)
TODO: Integration test's with facebooks internal DELETE
Contributor checklist
Release Notes
Please follow release notes guidelines and fill in the release notes below.