Skip to content

Ensure Substrait consumer can handle expressions in VirtualTable #16363

Open
@lorenarosati

Description

@lorenarosati

Describe the bug

When we convert a Substrait plan (which includes a virtual table) to a DataFusion LogicalPlan using the from_substrait_plan() function, the function from_read_rel() is called for the virtual table read. In this function, if the values field in the provided VirtualTable struct is empty, the read returns an empty relation. However, the values field of VirtualTable is deprecated and the expressions field of VirtualTable should be used to specify rows of a table; this function returns an empty table prematurely, when expressions is non-empty.

Returns EmptyRelation if values is empty:

Image
expressions should be supported in addition to the deprecated field values:

Image

To Reproduce

  1. Build a Substrait plan with expressions in embedded virtual table.
  • Here's a snippet of a Substrait plan in YAML format with a virtual table that has expressions but no values.
    Image
  1. Convert the Substrait plan to a DataFusion LogicalPlan and execute the query.
  2. Observe that the result has 0 rows.

Expected behavior

from_read_rel() should not return an empty relation and there should be rows in the outputted table.

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions