kernel meeting notes 2025-05-15

[zachschuermann]

Summary

crate split paused. 0.11.0 upcoming

ongoing threads

• IN/NOT IN PR to merge @roeap
• predicate/expressions split @scovich - merging lots of PRs
• merge opaque expressions @scovich
• merge IN/NOT IN deprecation @zachschuermann
• scan incremental API from @roeap
• @hntd187 targeting CDF reads, need timestamp work
• two PRs for checkpoint write support from @sebastiantia
• #[derive(IntoEngineData)] (zach)
• CREATE TABLE (zach)
• stats for writes (doing simple non-default engine for now)
• domain metadata (reads) (zach)
• need row id's (re-introduce predicate pushdown in
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• robert: working on support for multiple object stores
• TODO prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• test only sync engine
• default engine execute

Attendees

@zachschuermann @nicklan @mightyshazam @hntd187 @OussamaSaoudi @roeap @scovich @rtyler

notes stream

• upcoming: 0.11.0 for checkpoint writes, predicate split
• steve: CDF read integration with kernel removing 1000LOC+
• write FFI inbound
• MSRV 1.82 update
• zach: splitting crates update
  • Error framework refactor: error codes (database engines, rustc, etc. do this)
  • orphan rule (TryFrom)
• zach: remove global scan state
  • TODO prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• robert: from datafusion - we have logical schema-based predicate, how to translate to physical predicate for pushdown?
  • we associate a physical predicate already?
  • when you create scan you pass a predicate
  • then we translate to physical predicate (might just need to expose?)
  • if predicate is static (always accept/reject) it understands that
• zach: remove scan::execute - put better one in default engine
  • note that we use execute a lot
  • proposal move in SyncEngine
  • we don't want a public API that gives users sense of entitlement of performance
  • follow up: syncengine test only? yes!
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory
• zach: CRC reads
• zach: stats for writes
• robert's datafusion integration
• zach: prioritize DAT more?
• ryan: thoughts on predicate split/simplification
• tyler: HDFS toil, cargo features need to be tested! need basic smoke tests: if you have S3 object store make sure we can read tables.
• integration tests needed! run with real world. (not just in mem)
• HDFS tests for PRs (separate job in CI)
• ryan: why do we need HDFS? currently only in tests -- but tyler made stuff configurable now so really no need to specifically test HDFS (let's just test a custom object store)
• robert: S3 for testing against object stores
• SOLUTION: remove HDFS dep entirely! (tyler taking this!)
• delta-io is a paid for org in github (could do provisioned runners/self-hosted)
• CI improvemented? tyler poking around (better runners etc.)
• ryan: question on HDFS not related to build times, but why would kernel care about HDFS?