2025-06-05 kernel meeting notes

[zachschuermann]

Summary

0.12.0 released, and 0.12.1 (probably) coming very soon! (fixing azure issues)

current PRs

• IN/NOT IN PR to merge @roeap (waiting on opaque expressions. then will go in to default engine)
• !!!! opaque expressions @scovich
  • arrow opaque expressions need another level of indirection
  • need an 'arrow expression'
• #[derive(IntoEngineData)] (zach)
• HDFS (check with tyler) removal
• bunch of timestamp work from @OussamaSaoudi

POC/waiting PRs

• CREATE TABLE (zach)
• Crate split (deprio)

upcoming

• zach: metrics PR!
• need row id's (re-introduce predicate pushdown)
• @hntd187 targeting CDF reads, need timestamp work
• stats for writes (doing simple non-default engine for now): per-table write metadata schema
• CRC P+M optimization on reads
• kyjah: ask for SHALLOW CLONE in kernel - size/scope?
• prefactor: remove a scan-wide physical schema and push down to something that can access parquet footer
• remove execute, do default engine execute
• robert: working on support for multiple object stores?
• zach: prioritize DAT more?
• zach: FileIO/Evaluation split
  • nick/ryan: we could colocate EvaluationHandler stuff as a CPU-bound zone (and FileIO layer all IO-bound/blocking)
  • maybe call out: EngineData should be in memory

Attendees

@zachschuermann @nicklan @scovich @mightyshazam @joonyoo181 @hntd187 @dennyglee

notes stream

ongoing: azure issues

• original issue, can't do suffix range requests: fix: remove azure suffix range request #1006
• new: for some reason we have issues with CONTAINER name in path, see https:// URLs are broken on Azure #1007

some good real-time debugging!

• we are broken on https:// due to CONTAINER being in path, abfss:// works (explains how duckDB et. al. still work)
• zach to take fix

file issue against object_store on concurrent modification thing: if file changes size between HEAD and GET -> basically get UB

• would need to pass in etag or something to identify the file@version
• if you replace object, modtime will change --> could use modtime as etag proxy (we already have modtime in kernel)

as soon as azure stuff is fixed, will do 0.12.1 release.

open lakehouse mini-summit!!

still need to revisit metrics discussion from last time

metrics are hard

• careful: kernel can't aggregate metrics from batches bc e.g. you could run batches in parallel! then "addition" isn't correct.
  • counts could be easy?
  • duration is hard - scheduling. IO/wall clock time? etc.
• pass context to engine for them to populate
• context + how to accumulate?