2024-03-07 kernel meeting notes

[zachschuermann]

summary

lots of PRs in flight - still marching towards opening repo fully! nick's big PR landed. Also relaxing requirement on internal/external reviews: any two reviews to merge a PR in order to prioritize speed. discussion of delta-kernel and delta-rs integration.

action items

some PRs in flight:

• Switch to new data passing API #109
• start implementing FFI #91
• Fixed file path resolution so tests pass on Windows #131
• remove arrow dependency from data skipping #132
• Initial work to use maturin+pyo3 to get a python interface #130

attendees

@nicklan @roeap @zachschuermann @vkorukanti @tdas @MrPowers

notes stream

• nick exploring python API
• delta-rs string expressions: DNF for easy parsing. why not pyarrow?
• considerations for python API
• kernel/delta-rs have significant overlap in this regard
• likely only need one implementation
• robert: delta-rs could evolve into a datafusion delta implementation
• robert: don't maintain two python bindings for delta. consider publishing new bindings under deltalake package and just maintain one
• delta-sharing-pandas: how DV tables will be read in OSS
• python shim on top of kernel to enable lightweight reads of DV tables in python ecosystem (delta sharing pandas etc.)
• can happen with (1) rust kernel or (2) deltalake python package built on top of delta-rs. either
  • on kernel directly (1)
  • or delta-rs on kernel (2)
• robert: work on delta-rs for moving to API that kernel can plug in to
• td: should we invest time into delta-rs migration to kernel?
• robert: want to use many of the features in kernel in delta-rs.
• nick: one PR for notices file needs stamp. Add NOTICE file and cargo-about toml/hbs #133
• nick: earlier goal of one internal and one external review has proved difficult (and don't want to make robert the single bottleneck)
  • proposal: continue to require two reviews, but can merge with any two reviews in order to prioritize speed
  • robert: SGTM!
• robert: one thing to aim for in terms of contributors is to consolidate and keep community focused. community can get excited :)
• delta-rs planning is a large open question/lots of planning/design
• robert: there are a few people eager to become involved. excellent OSS rust engineers. will pull them in after 'full' open-sourcing
• td: jvm kernel doing conflict resolution, will propogate to rust kernel in the near future.
• balance moving fast for read support in delta-rs; can we architect an intermediate point in delta-rs which leverages kernel for reads but not writes?
• robert: conflict resolution already exists in delta-rs. already has write support. copied spark test suite.
• robert: one question is where repo will land after graduating from incubator. open question.