2025-05-08 kernel meeting notes

[zachschuermann]

Summary

released 0.10.0 with arrow 55 support! delta-rs upgraded to kernel 0.10.0 (on arrow-54 - making arrow 55 a later problem).

ongoing threads

• IN/NOT IN PR to merge @roeap
• predicate/expressions split @scovich - merging lots of PRs
• merge opaque expressions @scovich
• merge IN/NOT IN deprecation @zachschuermann
• scan incremental API from @roeap
• @hntd187 targeting CDF reads, need timestamp work
• two PRs for checkpoint write support from @sebastiantia
• #[derive(IntoEngineData)] (zach)
• CREATE TABLE (zach)
• stats for writes (blocked)
• domain metadata (zach)
• need row id's (re-introduce predicate pushdown in
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• robert: working on support for multiple object stores

Attendees

@zachschuermann @nicklan @mightyshazam @hntd187

notes stream

• upcoming: 0.11.0 for checkpoint writes, predicate split
• write FFI inbound
• MSRV 1.82 error
• Error framework refactor: error codes (database engines, rustc, etc. do this)
• zach: splitting crates - delta-kernel-engine crate (includes Engine impl) and delta_kernel crate
• zach: remove scan::execute? lots of read impls (examples, etc.)
  • nick: multi-thread in core kernel bad!
  • zach: remove from kernel, put in default engine (and make better)
  • ryan: reason to keep somewhere is for an example!
  • proposal: (1) example in core kernel (basically execute just for education), (2) pull out default engine execute and make better
  • execute a public part of kernel api? no.
  • should it exist as example? yes, as example in kernel.
  • third: do we want an 'easy button' execute-style API.
    • note we do have execution in default engine!
    • public API in default engine!
    • careful: this is same argument that got us scan::execute in the first place. solution: have to make it 'up to snuff'
    • make a 'correct' implementation without performance promises
• robert: incremental scan API
  • engine data vs wrap in struct
  • start this as internal_api
• ryan: VARIANT related question: normal struct: a.b.c - if query needs all columns, scan would receive a then later can pull out a.b.c. Query read schema is subset of table schema. with variant we still have a struct but there's shredding (possibly strongly typed fields). if you ask for v.b.c (v is variant) - v struct, extract b struct, extract c out of that (leaf). there's no guarantee that c exists or is even the right type. whenever reading underlying file, you could read a 'shredded' c and it's there, done. or it could be shredded partially and have a rescue column (incorrect type) or maybe no shredded c at all (or any parent above). reader must reconstruct everything from the pieces the scanner gives back (large tree of rescue columns, etc.) - can we prune v?
• nick: we create a scan with output schema, and can also pass a predicate. what if engine doesn't pass predicate.
• ex: SELECT v, v.b.c note this doesn't work today - but could it?
• SELECT v.b.c as INT, v.b.c as STRING yikes..
• proposal: change our semantics - instead of "you must have subset of table schema" we have "you pass a selection of table's schema (allow dups, etc.)" - an expansion of semantics (subset is trivially a selection)
• note this only really comes up for VARIANT
• UNION schema? but naming issues..