Skip to content

Migrate chunks on ingestion #9115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
emilk opened this issue Feb 24, 2025 · 0 comments
Open

Migrate chunks on ingestion #9115

emilk opened this issue Feb 24, 2025 · 0 comments
Assignees
Labels
enhancement New feature or request 👀 needs triage This issue needs to be triaged by the Rerun team

Comments

@emilk
Copy link
Member

emilk commented Feb 24, 2025

When loading a chunk containing old data, we want to migrate it by converting its contents to the latest format.
This probably mean using the old generated Rust component/datatype deserializer, having some custom transform run on it, and then writing it back with the new serializer. For archetypes it may require loading N columns and outputting M new ones.

For this to work we first need

What crate does this migration work? re_sorbet seems like a likely candidate, if we think it's fine that re_sorbet depend on re_types.

Initially we only need to support loading the two most recent version of data, not all previous versions.

In most cases this can be zero-copy.

Alternatives

We provide a migration tool, e.g. rerun migrate foo.rrd that updates an .rrd to the latest version. The downside is that this requires more work for users, and also won't work for on-wire data (e.g. coming from an older storage-node over gRPC).

Progress

@emilk emilk added enhancement New feature or request 👀 needs triage This issue needs to be triaged by the Rerun team labels Feb 24, 2025
@emilk emilk self-assigned this Feb 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request 👀 needs triage This issue needs to be triaged by the Rerun team
Projects
None yet
Development

No branches or pull requests

1 participant