execution: fix OOM crash due to db::Buffer::accounts realloc (#2064) #2081
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The flat_hash_map has a policy (in
rehash_and_grow_if_necessary()
) to grow 2x when its size reaches 25/32 (78%) of capacity.Previously the batch execution logic was adding things to db::Buffer, and then testing if its size becomes bigger than batch_size. This lead to significant overshooting, because at the growth threshold it will double the memory size and make it batch_size * 2 (even more, because there's also unused 22% of capacity).
In standalone silkworm execution stage the batch_size estimation was based on a gas-based heuristic formula, which lead to significant underestimation and an OOM crash when trying to allocate 4 Gb of RAM at once.
This changes the logic from checking batch_size aposteriori to apriori. If there's a possibility to grow the buffer significantly and overshoot the limit, we shouldn't take the risk, and commit the batch.
Additional changes: