Skip to content

Transfer encoded chunks from ingesters to querier in the blocks storage #3831

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pracucci opened this issue Feb 16, 2021 · 1 comment · Fixed by #3889
Closed

Transfer encoded chunks from ingesters to querier in the blocks storage #3831

pracucci opened this issue Feb 16, 2021 · 1 comment · Fixed by #3889
Labels
storage/blocks Blocks storage engine

Comments

@pracucci
Copy link
Contributor

Is your feature request related to a problem? Please describe.
The chunks storage was transferring encoded chunks from ingesters to querier at query time, while the blocks storage is transferring decoded ones. Having to decode chunks in the ingesters increases the memory utilisation at query time and increases the likelihood of getting ingesters OOMKilled because of an heavy query.

Describe the solution you'd like
Historically, TSDB didn't allow to query for chunks, reason why we had to transfer decoded time-series from ingesters to querier, but now it's possible and we should it.

@pracucci pracucci added the storage/blocks Blocks storage engine label Feb 16, 2021
@pracucci
Copy link
Contributor Author

Another side effect I've just learned about transferring decoded time-series from ingesters to querier is that protobuf unmarshalling is inefficient when transferring slices (repeated fields in protobuf). The problem is that, according to the doc, repeated fields don't carry the number of items in the serialized message and this prevents the unmarshalling from pre-allocating the slice with the right size when deserialising, causing a large number of growslice.

For example, this is the CPU profile in the querier while running an heavy query:

Screenshot 2021-02-16 at 10 53 31

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
storage/blocks Blocks storage engine
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant