2023-11-30 kernel meeting notes

[zachschuermann]

summary

@nicklan went over a new duckDB connector prototype and some FFI changes

action items:

• review list of immediate actions and put in Public release goals #42
• nick design doc
• new hackathon/FFI code review

attendees

@wjones127 @nicklan @MrPowers @zachschuermann @ryan-johnson-databricks @tdas @roeap

notes stream

[started taking notes late - will fill in more later]

@nicklan duckDB delta kernel hackathon

• TLDR duckDB extension implemented via kernel. exposes scan_delta(path) TVF
• learnings
  • constraining col batch: make the col vector/batch opaque, expose minimal visitor interface
  • something to follow up later: endianness?
  • agreement between engine/connector and kernel: primitive types
    • design doc will include maps/structs/arrays considerations
    • note: we can have maps of arrays etc. but we don't need those in kernel. that is, the delta spec doesn't need some crazy nested types. we just need string -> string map, array string, etc.
    • if we do this 'visitor' stuff, we might not even need to support array/map/struct in the sense of returning complex values, but we just need to be able to 'extract' the key/value/leaf/etc. which is just a primitive type
    • should consider expression handler must be able to 'speak' connector data type
    • consider: default expr handler will be built on top of arrow
    • if you want to use this default expr handler you will have to speak arrow!
  • uncovered interesting quirk of async implementation
    • if you have ref to table client (or rather reader)
    • reader holds on to executor ref
    • you can drop that before we go over to the iterator (the receiver end of the channel)
    • then you drop the executor and all the futures get cancelled!
    • TODO need to think about async client and how to keep executor alive (for now just keeping reader in Arc and you retain active ref)
• (action item) sharing a branch, currently just hackathon code but would like to get feedback and start merging pieces!
• @wjones127 thought: This might make the expression handler much more complex to implement
  • Colbatch is big API
  • if we move to opaque col batch, then we move the lift to expression handler?
  • have to build the 'extraction' etc.
  • (also please tell me if i'm capturing this correctly)
  • nested field access might be an afterthought in some query engines (in @wjones127 experience)
  • flag: we probably want to have standardized test suite
  • for each expression handler we want to flex all the different patterns that kernel uses
  • @ryan-johnson-databricks: nested access is potentially a big lift for client/connector?
    • the opaque column blob might not 'solve' the API complexity but rather just move it to expression
    • (this is the hypothesis)
  • consideration: all custom and all default engine client are easier, if someone wants to mix-and-match it is likely more difficult.
  • trait for something? impl for each?
• @roeap doing some work on delta-rs side
  • using kernel abstractions?
  • checkpointing: do we need fanciness if there are ~10 diffs? considerations: checkpointing happens every 100 sometimes :) and there are some times with many more/large commits
  • for P+M replay: should we remember other commits? Risk of OOM. More complexity. Is it worth it?
  • initial reaction: let client cache parsed json if they wish. default client can do something sensible. in general it makes sense to not duplicate parsing or go back to cloud store.