Skip to content

datastore: .ef v2 format #2861

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 20, 2025
Merged

datastore: .ef v2 format #2861

merged 2 commits into from
May 20, 2025

Conversation

battlmonstr
Copy link
Contributor

@battlmonstr battlmonstr commented Apr 16, 2025

  • added new snapshots encoder method: decode_word_with_metadata(); it is needed so that the new codec has info about the base timestamp of the file before decoding
  • added a new type InvertedIndexTimestampList to represent II values, and refactored II queries to work with it instead of EliasFanoList32 directly
  • InvertedIndexTimestampListDecoder implements the new encoding format
  • refactored EliasFanoList32 data_holder_ to be the primary source of truth; this simplifies InvertedIndexTimestampListDecoder code, from_encoded_data overloads code and it is a step towards fixing the dangling span after move problem
  • added step_size/make_step_converter to the repo schema so that bundles can be created with a proper step_converter
  • elias_fano_decoder_test.cpp is merged into elias_fano_list_test.cpp

@battlmonstr battlmonstr force-pushed the pr/ii_v2 branch 2 times, most recently from af339d5 to 87d512f Compare April 23, 2025 09:44
@battlmonstr battlmonstr force-pushed the pr/ii_v2 branch 2 times, most recently from e0dbc2f to 6fa1f26 Compare April 28, 2025 09:10
@battlmonstr battlmonstr requested a review from canepat April 28, 2025 09:24
@battlmonstr battlmonstr added the snapshots Framework for BitTorrent-based snapshots label Apr 28, 2025
@battlmonstr battlmonstr marked this pull request as ready for review April 28, 2025 09:24
@battlmonstr battlmonstr force-pushed the pr/ii_v2 branch 3 times, most recently from 4143415 to b7a1621 Compare April 30, 2025 06:53
@canepat canepat added the erigon3 Erigon3 feature label Apr 30, 2025
@canepat canepat merged commit 41185c3 into master May 20, 2025
5 checks passed
@canepat canepat deleted the pr/ii_v2 branch May 20, 2025 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
erigon3 Erigon3 feature snapshots Framework for BitTorrent-based snapshots
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants