2025-03-06 kernel meeting notes

[zachschuermann]

Summary

0.8.0 released!

Pull requests/ongoing threads

• Change the discussion format, particularly the pull requests/ongoing threads section.
• CI to build nightly with external integrations and make sure FFI works
• Upcoming priorities for kernel
• create_one API
• col [NOT] IN API
• snapshot incremental API: Snapshot::new_from(Arc<Snapshot>) -> Arc<Snapshot>
• @hntd187 targeting CDF reads, need timestamp work

Attendees

@nicklan @sebastiantia @hntd187 @roeap @zachschuermann @mightyshazam

notes stream

• 0.8.0 release
  • JSON reader improvement
  • new FFI for partition columns on Snapshot not Scan
• v2checkpoint
• variant
  • semi-structured. two columns of binary
  • if you have those columns, something in kernel to transform into something understandable by the engine
  • kyjah: no needs right now for .NET side; systems that want this have first-class JSON type. .NET has lots of ORM usage
• caching (foyer.rs)
  • ryan: what are we caching? raw bytes from cloud? parsed json? memorized log replay?
  • robert: delta-rs currently keeping entire log replay in memory. delta-rs snapshot exposes txn actions. Streaming/ingestion commit heavy workloads relying on this.
  • idea: mitigate perf hit (where?) at JSON read level.
  • robert: json reader caching parsed arrow (feather?)
  • should default-engine have this?
  • is this the right performance solution? two ways to have duplicate requests that caching could help:
    1. multiple log replay paths (PM, CRC, app id, full one) - does parsing help here? maybe raw bytes would be best (we push schema into the json read)
    2. for log replay: want to reuse log replay but can't because data skipping predicates are different every time. could do non-skipping log replay, or just go back to json few times (bc of amount of pruning). big tradeoff (one inefficient scan that can't be reused, or many efficient scans that can be reused). cache parsed json before we parse the stats, bc the stats are string of json literal (nested) - need to cache between those two steps bc the second layer is schema-specific
  • robert use case: app id's
  • there are arguments to be made for caching at many different levels (bytes, parsed, log replay, etc.)
  • robert proposal: raw bytes caching in default-engine? limited to JSON reads? Parquet side doesn't get benefit?
  • note: default-engine can be handed object store implementation
  • can a CachingObjectStore or something help here?
  • for robert: if IO isn't repeated, what is the worst regression here (for delta-rs integration)?
  • concern: lots of JSON to read.