Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update begin delete #24528

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

shelton408
Copy link

@shelton408 shelton408 commented Feb 11, 2025

Description

Changes to separate ConnectorTableHandles for deletes into it's own interface (ConnectorDeleteTableHandle) similar to ConnectorInsertTableHandle

If you have a connector that is broken by these changes, please refer to the fixes to presto-iceberg as an example. If you do not implement beginDelete, the changes will not affect you.

Motivation and Context

Please refer to #24520

Impact

Metadata, ConnectorMetadata, MetadataManager changed
->downstream affects execution: TableWriteInfo, ExecutionWriterTarget, TableWriterNode
Create 2 new types: ConnectorDeleteTableHandle, DeleteTableHandle
Types Changed: TableWriterNode.DeleteHandle

Also affects finishDelete, getDeleteRowIdColumnHandle types (AbstractMockMetadata, DelegatingMetadataManager)

Other classes affected:
ConnectorPageSinkProvider -> provide new pagesink for deletes?
ConnectorHandleResolver -> provide new getDeleteTableHandleClass method -> HandleResolver implements new ConnectorDeleteTableHandle

All connectors affected, those that implement DELETE will require more changes.
Kudu and iceberg connector both need to change types used for begin/finish delete
Iceberg currently uses IcebergTableHandle which inherits from basehivetablehandle->connectorTableHandle, and we would need to create a separate IcebergTableHandle for deletes

This would eventually apply to update as well if were going to make the code symmetric

Test Plan

Pass presto-icebergs tests (includes DELETE tests)
TODO: Integration test's with facebooks internal DELETE

Contributor checklist

  • Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* Fix OSS connectors affected by changes 
* Add serialization for new types
* Add pagesink for DELETES to support future use
* Update beginDelete to return the new types, and finishDelete to accept the new types

SPI Changes
* Add a separate ConnectorDeleteTableHandle interface for `ConnectorMetadata.beginDelete` and `ConnectorMetadata.finishDelete`, replacing the previous usage of ConnectorTableHandle
* Add DeleteTableHandle support these changes in Metadata

Hive Connector Changes
* Replaced return type of beginDelete

Iceberg Connector Changes
* Fix IcebergTableHandle implementation to work with new types used in begin/finishDelete

Kudu Connector Changes
* Replaced return type of beginDelete

@shelton408 shelton408 requested a review from a team as a code owner February 11, 2025 01:42
Copy link

linux-foundation-easycla bot commented Feb 11, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

@shelton408
Copy link
Author

Has passed presto-Iceberg delete tests In the current state

steveburnett
steveburnett previously approved these changes Feb 17, 2025
Copy link
Contributor

@steveburnett steveburnett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! (docs)

Pull branch, local doc build, looks good. Thanks!

ZacBlanco
ZacBlanco previously approved these changes Feb 18, 2025
Copy link
Contributor

@ZacBlanco ZacBlanco left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The changes on the Iceberg side look good to me

@ghelmling
Copy link
Contributor

Can you update the PR description with a release note, something like:

== RELEASE NOTES ==

SPI Changes
* Add a separate ConnectorDeleteTableHandle interface for `ConnectorMetadata.beginDelete` and `ConnectorMetadata.finishDelete`, replacing the previous usage of ConnectorTableHandle.

Copy link
Contributor

@ghelmling ghelmling left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a couple minor notes on comments to clean up.

For the "Release Notes", I think you only really need the "SPI Changes" section and I'm not sure you need to mention DeleteTableHandle as the usage seems internal instead of part of the Connector SPI. You could instead list a second bullet on the ConnectorHandleResolver change that was also needed.

@rschlussel
Copy link
Contributor

Kudu tests are failing. I expect you need to update the KuduHandleResolver similar to Iceberg.

Changes to update finishDelete to new types

Fixed Iceberg Delete as proof of concept. Fixed missing serialization changes.

Kudu fix and checkstyle

recombine iceberg tablehandle and deletetablehandle, fix serialization

Documentation and add delete pagesink

Fix implementation of delete pagesink

delete extra comments/changes

update KuduTableHandle

remove comments and unused handle

add kudu resolver for delete handle
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants