[bitnami/thanos] add support for sharding in storegateway

[mhulscher]

Which chart:

bitnami/thanos

Is your feature request related to a problem? Please describe.

By default, the thanos-storegateway will advertise that it can serve all timeseries and all blocks in the object store (ie S3-bucket). As such, you can not scale the number of stores and register them all with thanos-query. Thanos-query will complain about multiple stores that advertise the same data with non-unique labels. The storegateway can be put behind a loadbalancer, such as a kubernetes Service (of type ClusterIP), but the fact that communication happens over long lived gRPC connections means that, most of the time, there is no guarantee that each storegateway replica will serve an equal amount of requests.

This is a known issue and documented on the thanos website and can be solved by introducing sharding (aka partitioning) in the storegateway. Sharding allows each replica of the storegateway to only serve some blocks but not others. A request that spans multiple blocks will then automatically be fanned out to multiple stores.

Describe the solution you'd like

Introduce the configuration of optional time-based or hash-based partitioning (each nth-block). Default the store to use a single partition. The storegateway could be converted to a deployment with an index suffix, like so:

desc	deployment name
first partition	thanos-storegateway-0
second partition	thanos-storegateway-1
third partition	thanos-storegateway-2

Store could also remain a statefulset, although I am not sure why it is a statefulset at the moment.

Introducing a partition suffix to the store statefulset/deployment will cause it to be re-created. This could be considered a backwards incompatible change.

Describe alternatives you've considered

Use the thanos chart from banzai-cloud, which already supports sharding. The banzai chart seems less actively maintained.
https://github.com/banzaicloud/banzai-charts/blob/master/thanos/values.yaml#L174

Additional context

desc	link
Official documentation	https://thanos.io/tip/thanos/sharding.md/
CNCF slack discussion in #thanos	https://cloud-native.slack.com/archives/CK5RSSC10/p1604923594031700
Example implementation	https://github.com/observatorium/deployments/blob/bf1304b0d7bce2ae3fefa80412bb358f9aa176fb/environments/openshift/manifests/observatorium-template.yaml#L1514-L1521

[andresbono]

Hi @mhulscher, thanks for opening this issue. At a first sight, I don't see any cons to extend the functionality of our chart with sharding capabilities.

Would you like to contribute with a PR addressing the changes that you are proposing? We will be more than happy to review it.

[stale]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[stale]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[psandhu79]

Hi

just wondering if anyone will pick this up ?

Or is a straight copy from the banzai charts

[marcosbc]

Hi @psandhu79, note that we're open for contributions. If you are able to implement this feature in the Thanos chart, feel free to send a PR! We'd be glad to help with the review and release processes.

[psandhu79]

@marcosbc @andresbono I will see what i can do, one key question for me from the description above

Store could also remain a statefulset, although I am not sure why it is a statefulset at the moment.

Can i convert this to a deployment set ?
I have introduce an index this will backwards incompatible change is that ok ?

[marcosbc]

@psandhu79 Note that if you introduce a breaking change, the major version of the chart will need to be increased and the upgrade process should be documented in the "Upgrade" section of the README.

That said, why specifically convert from statefulset to deployment? Is there some limitation in the statefulset that would be solved by changing to deployment?

[psandhu79]

@marcosbc i have made the contribution awaiting final review and merge