Skip to content

bug: iceberg sink precompute partition key would generate wrong data if table partition key changed #18185

@chenzl25

Description

@chenzl25

In #14710, we support:

support get partition info at frontend
add the project node to compute the partition value and exchange node to shuffle by it

However, if the iceberg table partition key has been changed, iceberg sink will use the wrong partition key, or even worse it might generate data files that don't match the partition spec. Iceberg's hidden partitioning So I think we should only use the partition spec of creation to shuffle data, but when we write data to iceberg table, we should leverage iceberg's hidden partition to deal with the partition key changed problem.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions