2023-10-19 kernel meeting notes

[zachschuermann]

summary

Lightweight sync then chatted about #26

action items:

• expression PR review

attendees

@rtyler @wjones127 @nicklan @MrPowers @zachschuermann @vkorukanti @roeap @ryan-johnson-databricks

notes stream

• some discussion with AWS on C++ consumer of kernel. we should hear from them again in a month? aligning desired API
• useful to show them a (future) DuckDB connector

in-flight PRs

• feat: support more expression types #26
• feat: provide basic columnar batch API #41
• Traits for everyone #51

from @roeap: dagster in prod!

nullability handled in types themselves vs systems that use the types. nullability should be expressed as Expression::Null and that's it. should we have 'typed' null? e.g. the difference between u32 Null and u64 Null.

expression evaluation: should probably enforce expected return schema? only take input schema https://github.com/delta-incubator/delta-kernel-rs/blob/main/kernel/src/lib.rs#L112

if we take input/output schema: how disruptive would that be? if we get to a point where we (kernel) cannot determine output schema that's a red flag. for now: col pruning, stats/file skipping, DVs, row_id generation, constraints, column mapping, actual query schema we handoff: engine should tell us the schema. all prior examples have predetermined output schema. from @wjones127 we could have API be evaluate and get_output_schema. from @ryan-johnson-databricks we can do both (kernel/connector schema validation).

expressions

pub enum Expression {
    // Null, ?
    Literal(Scalar),
    Column(String),
    BinaryOperation {
        op: BinaryOperator,
        left: Box<Expression>,
        right: Box<Expression>,
    },
    UnaryOperation {
        op: UnaryOperator,
        expr: Box<Expression>,
    },
}

pub enum BinaryOperator {
    // Logical
    And, // binary or unbounded (aka AND a Vec)?
    Or,
    // Arithmetic
    Plus,
    Minus,
    Multiply,
    Divide,
    // Comparison
    LessThan,
    LessThanOrEqual,
    GreaterThan,
    GreaterThanOrEqual,
    Equal,
    NotEqual,
}

pub enum UnaryOperator {
    Not,
    IsNull,
}