2025-02-06 kernel meeting notes

[OussamaSaoudi]

Summary

Pull requests/ongoing threads

Attendees

@nicklan @zachschuermann @OussamaSaoudi-db @roeap @mightyshazam @sebastiantia @scovich

Leftovers

• Change the discussion format, particularly the pull requests/ongoing threads section.
• CI to build nightly with external integrations and make sure FFI works
• Upcoming priorities for kernel
• create_one API
• col [NOT] IN API
• snapshot incremental API

notes stream

• @roeap Goal this quarter is to improve conflict resolution. Ideally we can construct different states to test the logic.
• @OussamaSaoudi deterministic simulation testing?
• @scovich DST +1 for conflict resolution and data racing. spark hit transaction serializability bugs. Use random testing and customer input to find cases, but DST should encode these cases so that we never regress on this bug. If there are cases we know that break
• @OussamaSaoudi How much tracing do you need
• @scovich if the. protocol is broken, a reasonable amount of tracing should be able to easily find bugs. Ex; Non-repeatable reads. Other cases came from thread races. These are tough to find, even with tracing turned on.
• @roeap QP: original author of delta-rs. There's interesting frameworks for proving distributed algorithms. TLA+
• @OussamaSaoudi stateright is also used. It exhaustively checks your algorithm
• @nicklan has it proven useful?
• @roeap It was used to prove the S3 locking based on dynamo-db locking.
• @nicklan RRR is a distributed debugger that's interesting
• @nicklan The changes for expression-based transforms are merged on the rust side. FFI side is in review 🎉
• @mightyshazam Is there a plan to implement an FFI engine interface? delta-rs requires a rust engine.
• @nicklan We (kernel) definitely want to enable an engine implementation in other languages. Are there any blockers to achieving that. Primarily passes around function pointers that kernel can then use/access.
• @mightyshazam The dyn Engine requires that you implement the engine trait in rust.
• @scovich The rust bridge for FFI (introducing engine trait to FFI engines) is WIP. It's a matter of writing the bridge code. It's significant amount of work
• @nicklan ExternEngineVTable is what we'll use for implementing the trait using just a C FFI struct.
• @scovich The dyn/Arc becomes rly ugly and requires another bridge due to Unsized types. There's also the issue of mixing engines. We need a way to detect which engine is which. As soon as you implement your own EngineData, you need to implement your own expression evaluator.
• @mightyshazam Ran into this issue and came upon a lot of work
• @scovich Engine has to write a lot of callbacks. We want to eventually enable incremental implementation of all the callbacks.
• @mightyshazam Everything is in arrow, but dotnet's ecosystem is a lot richer for object store. Parquet-dot-net would be nicer and would spark wider adoption.
• @zachschuermann Is the in-memory format for all these arrow?
• @mightyshazam Arrow would be used to move data between these
• @nicklan It should be easy to move arrow from ffi and use the default engine by wrapping the arrow data.
• @mightyshazam Passing enginedata is not a problem. Anything that takes an extern engine, it becomes a headache.
• @zachschuermann Ideally, dot-net can use arrow-expression, but only displacing the object store or filesystem implementation in the default engine.
• @nicklan If we expose more, we can allow ppl to swap out the parquet and json handlers for their own, but keeping the expression evaluator. It's assumed that the engine's handler's data should be interpretable by the expression evaluator.
• @scovich right, the engine is assuming that the data came from the right place
• @nicklan There's no actual problem using different pieces, as long as the data is mutally interpretable.
• @scovich Just need to make sure that when you create engine data, make it the arrow engine data that the default engine uses. Then just reuse the default engine's implementation for row visitors and expression evaluation.
• @mightyshazam THat was plan B. Enable the default engine feature, then don't use default engine fully
• @scovich Its reasonable to ask the arrow expression evaluation to be its own separate feature so that others can use it
• @mightyshazam There's a lot of coupling in the arrow util folder. There might need to be more options to granularly separate it
• @scovich We can also put in the work to split them. We had to move the arrow stuff to a separate module in the past, so we can do it again to split apart the expression evaluator.
• @zachschuermann create-one is in flight. Two approaches: dummy data and evaluate expression over it, or scalars and put in schema
• @zachschuermann Col IN [list]
• @zachschuermann On snapshot update, prefer an immutable approach Snapshot::new_from
• @scovich delta-spark made a bad design choice in the past regarding snapshot::update. there are ~200 places in the code that do this snapshot update. It's a mess. Goal is to leave the engine in control. The engine should be the one to decide whether we refresh the snapshot
• @roeap Talked with zach, left comments on the PR. Internally mutating snapshots may not be the best approach. Consuming self ensures we don't have a bunch of snapshots lying around.
• @roeap What he mentioned new-from, creating a new snapshot in rust may not be dropped across threads, but engine may hold its own version of a snapshot
• @scovich engine may want to repeatedly reada snapshot for a transaction. So we don't want to consume the snapshot. We may need copying, or Arc if copying is too expensive.
• @roeap Saving snapshot isn't much since we just save the P&M and log segment. But in delta-rs, we keep the files around and replay the snapshot repeatedly.
• @scovich It wasn't that delta-rs kept files around, but you'd already done a log replay, and you wanted to run it through again without replaying through the log again.
• @scovich We may need to split up the API so engine can inject into that point
• @roeap It shouldn't only consume record batches that are log shaped, but also scanFile shaped
• @scovich It's reasonable for th eengine to do that without knowing what's in the scanfile. Engines cache stuff, and they should cache stuff. We're not breaking abstractions by letting the engine cache stuff and serve it agian.
• @roeap After that it was easy. Here's data, comes from a checkpoint, and (caching?) just works.
• @nicklan Kyjah, as you come up on issues, pls bring it up or ping us on slack.
• @scovich Filing issues preferred since it's easier to persist and review. This is important work that we don't want to drop
• @mightyshazam Trying to avoid duplicate efforts. I can work around it, but if you're worknig on it, that's great. I can provide more context as I work on it. You'll either see a PR to fix it, or an issue to kernel.
• @nicklan ty for engaging with this early work!! pls file issues. You can also solve issues.
• @scovich We've had ppl pick up issues, so that can get this stuff out the door