2025-01-23 kernel meeting notes

[OussamaSaoudi]

summary

• Using the discussion as a way to detect blockers in community PRs.
• Discussing where vacuum/optimize belong (whether in the kernel, or outside of it).
• The new arrow feature flags that are coming will help integrations

pull requests/ongoing threads

• @nicklan Introducing Expression Transforms Parts 2-4
• @timsaucer Partition column data skipping
• @zachschuermann Transaction and append data FFI
• @OussamaSaoudi Schema compatibility check + applying to CDF
• @OussamaSaoudi working on TableConfiguration
• @rtyler Introducing feature flags to handle arrow
• @kssenii Fixing schema ffi nullability and adding metadata field to schema
• @roeap Snapshot::update to reuse/extend existing log segment
• @roeap Support for col IN (a, b, c) expressions

attendees

@nicklan @hntd187 @zachschuermann @OussamaSaoudi-db @roeap @mightyshazam @dennyglee @sebastiantia

leftovers

• Change the discussion format, particularly the pull requests/ongoing threads section.
• CI to build nightly with external integrations and make sure FFI works

notes stream

• zach: TODO(Oussama): for the discussion, list each contributor, what they're working on, and whether they're blocked. Instead of the current pull requests/ongoing threads format. Let people add their own updates prior to the meeting. we could also do this in the current meeting. Thoughts?
• nick: Lets give it a go. It ensures that community PRs don't languish. Good way for community members to ping core maintainers. We need a way to communicate that to community.
• Zach: We could try to make an announcement and make it discoverable.
• nick: For ppl integrating, the expression transform work is starting to merge. Detailed instructions to migrate to expression transform is incoming (within next few weeks). It changes flow for connectors that read partitioned columns. No longer using a map of partition columns, we can give you the expression. Then you either apply it, or advanced parquet readers can insert constant columns as part of the parquet file.
• Denny: Pypi package source distribution for kernel.
• nick: We have no python wheel.
• denny: the delta-sharing wrapper
• nick: do we want to publish that? I'll have a look, but that's not part of kernel. This is maintained by the delta-sharing folks. For the time being, kernel is not venturing into the python ecosystem. Delta-rs provides a lot of these bindings already, so we need no divy up the responsibilities.
• robert: We can get a lot of reach with python bindings. Delta-sharing is just another way to construct backend. This is just another way that we can construct snapshot. By adopting kernel, delta-rs could be the defacto for delta-sharing. On credentials, we should be able to work with most venders at this point.
• zach: any blockers or questions? One more thing to cover: Any feature requests or expectations?
• Kyjah: Request: full parity with delta-spark over FFI. In reality, being able to do checkpoint, optimize and vacuum would let them move from delta-rs to kernel. (this is for delta-dot-net)
• Robert: Vacuum and optimize require higher level sql logic that kernel doesn't want to implement. This lives in the engine territory.
• Kyjah: Vacuum and checkpoints are what I really want.
• Robert: checkpoints are definitely kernel.
• zach: vacuum and optimize are engine responsibility. Using delta-rs for that makes sense. You can also do your own arrow-based compute to do your own optimize or vacuum. For checkpoints, we will only write v2 (currently building). v1 checkpoint writes will not be supported. This is coming in the following weeks.
• Kyjah: On vacuum, I just need removing old checkpoints instead of removing tombstones.
• zach: Vacuum combines a lot of stuff. Vacuum consists of log file cleanup and removing old checkpoints. Ideally, they should be two separate commands, but they're hidden behind vacuum. Story for vacuum is incoming. We'll make a decision on how to handle the different parts of vacuum. Deleting parquet files is engine duty. Log file cleanup and returning those log files sounds like a kernel responsibility. Since this is an ask, we can start prioritizing it.
• robert: Working on adopting log replay in delta-rs. Was looking into vacuum operation and do the planning based on the kernel expression system. We currently iterate over log, but planning should be done on arrow array. For vacuum, current expression system should be capable of doing that. No joins or complex operations are needed for vacuum. Just gets the list of files to delete. If we can express this as expressions, it's feasible to build in kernel.
• Kyjah: When adding functionality to kernel, make sure to expose it to FFI. Goal is to keep delta-dotnet's feature parity with the kernel. Happy to help with developing the FFI.
• nick: delta-dotnet will have 3 major integrating issues: 1) needs to be parity in the FFI layer. It's easy for FFI to be forgotten. Automatic detection will be hard. FFI is first class and is considered in the design.
• Kyjah: For delta-rs, I had to write all the FFI and structs. Maybe setup CI to run against main nightly.
• nick: It would be great to have nightly CI from other projects. We've talked about having nightly CI that builds known integrations with kernel and ensures they're still working. Github actions are annoying, so it'll come later. We'll continue working on FFI
• nick: On integration, tyler has a PR to change arrow support to feature flags. This is important to integration folks. Cargo isn't good enough for dependency resolution we want. Feature flags seems to be standard within the ecosystem. Just enable the feature flag for the version of arrow you want. We'll aim to support as many versions of arrow as we can. Other nice thing: Could use conditional compilation for small differences in arrow versions. Goal is to maintain backwards compatibility for arrow versions