Skip to content

Allow commitInfo's operationParameters to have non-string values #501

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zachschuermann opened this issue Nov 15, 2024 · 1 comment
Open

Comments

@zachschuermann
Copy link
Collaborator

zachschuermann commented Nov 15, 2024

For existing tables in the wild we've seen some with "operationParameters":{"version":10,"timestamp":null}. This breaks reads with the following error:

Error: Arrow(JsonError("whilst decoding field 'commitInfo': whilst decoding field 'operationParameters': expected string got 10"))

Despite commitInfo allowing arbitrary JSON (unless constrained by in-commit timestamps), we attempt to prescribe structure when reading. This should almost never fail but due to arrow json reading it errors on attempting to read int 10 as a String.

We likely need to come up with a way to allow deserializing arbitrary JSON in operationParameters? How can we represent that with our existing Schema system?

@zachschuermann
Copy link
Collaborator Author

related: delta-io/delta#2419

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant