Skip to content

Commit 67c0985

Browse files
roystchiangroystchiang
authored
add memcached auto-discovery support from thanos (#4412)
* add memcached auto-discovery support from thanos (#4412) Signed-off-by: Roy Chiang <[email protected]> * update documentation Signed-off-by: Roy Chiang <[email protected]>
1 parent a085a4e commit 67c0985

File tree

15 files changed

+328
-27
lines changed

15 files changed

+328
-27
lines changed

‎CHANGELOG.md

+1
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@
4141
* [ENHANCEMENT] Exemplars are now emitted for all gRPC calls and many operations tracked by histograms. #4462
4242
* [ENHANCEMENT] New options `-server.http-listen-network` and `-server.grpc-listen-network` allow binding as 'tcp4' or 'tcp6'. #4462
4343
* [ENHANCEMENT] Rulers: Using shuffle sharding subring on GetRules API. #4466
44+
* [ENHANCEMENT] Support memcached auto-discovery via `auto-discovery` flag, introduced by thanos in https://github.com/thanos-io/thanos/pull/4487. Both AWS and Google Cloud memcached service support auto-discovery, which returns a list of nodes of the memcached cluster. #4412
4445
* [BUGFIX] Fixes a panic in the query-tee when comparing result. #4465
4546
* [BUGFIX] Frontend: Fixes @ modifier functions (start/end) when splitting queries by time. #4464
4647
* [BUGFIX] Compactor: compactor will no longer try to compact blocks that are already marked for deletion. Previously compactor would consider blocks marked for deletion within `-compactor.deletion-delay / 2` period as eligible for compaction. #4328

‎docs/blocks-storage/querier.md

+15
Original file line numberDiff line numberDiff line change
@@ -512,6 +512,11 @@ blocks_storage:
512512
# CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-item-size
513513
[max_item_size: <int> | default = 1048576]
514514
515+
# Use memcached auto-discovery mechanism provided by some cloud provider
516+
# like GCP and AWS
517+
# CLI flag: -blocks-storage.bucket-store.index-cache.memcached.auto-discovery
518+
[auto_discovery: <boolean> | default = false]
519+
515520
chunks_cache:
516521
# Backend for chunks cache, if not empty. Supported values: memcached.
517522
# CLI flag: -blocks-storage.bucket-store.chunks-cache.backend
@@ -559,6 +564,11 @@ blocks_storage:
559564
# CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-item-size
560565
[max_item_size: <int> | default = 1048576]
561566
567+
# Use memcached auto-discovery mechanism provided by some cloud provider
568+
# like GCP and AWS
569+
# CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.auto-discovery
570+
[auto_discovery: <boolean> | default = false]
571+
562572
# Size of each subrange that bucket object is split into for better
563573
# caching.
564574
# CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-size
@@ -625,6 +635,11 @@ blocks_storage:
625635
# CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size
626636
[max_item_size: <int> | default = 1048576]
627637
638+
# Use memcached auto-discovery mechanism provided by some cloud provider
639+
# like GCP and AWS
640+
# CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.auto-discovery
641+
[auto_discovery: <boolean> | default = false]
642+
628643
# How long to cache list of tenants in the bucket.
629644
# CLI flag: -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl
630645
[tenants_list_ttl: <duration> | default = 15m]

‎docs/blocks-storage/store-gateway.md

+15
Original file line numberDiff line numberDiff line change
@@ -576,6 +576,11 @@ blocks_storage:
576576
# CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-item-size
577577
[max_item_size: <int> | default = 1048576]
578578
579+
# Use memcached auto-discovery mechanism provided by some cloud provider
580+
# like GCP and AWS
581+
# CLI flag: -blocks-storage.bucket-store.index-cache.memcached.auto-discovery
582+
[auto_discovery: <boolean> | default = false]
583+
579584
chunks_cache:
580585
# Backend for chunks cache, if not empty. Supported values: memcached.
581586
# CLI flag: -blocks-storage.bucket-store.chunks-cache.backend
@@ -623,6 +628,11 @@ blocks_storage:
623628
# CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-item-size
624629
[max_item_size: <int> | default = 1048576]
625630
631+
# Use memcached auto-discovery mechanism provided by some cloud provider
632+
# like GCP and AWS
633+
# CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.auto-discovery
634+
[auto_discovery: <boolean> | default = false]
635+
626636
# Size of each subrange that bucket object is split into for better
627637
# caching.
628638
# CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-size
@@ -689,6 +699,11 @@ blocks_storage:
689699
# CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size
690700
[max_item_size: <int> | default = 1048576]
691701
702+
# Use memcached auto-discovery mechanism provided by some cloud provider
703+
# like GCP and AWS
704+
# CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.auto-discovery
705+
[auto_discovery: <boolean> | default = false]
706+
692707
# How long to cache list of tenants in the bucket.
693708
# CLI flag: -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl
694709
[tenants_list_ttl: <duration> | default = 15m]

‎docs/configuration/arguments.md

+2
Original file line numberDiff line numberDiff line change
@@ -533,6 +533,8 @@ The DNS service discovery, inspired from Thanos DNS SD, supports different disco
533533

534534
If **no prefix** is provided, the provided IP or hostname will be used straightaway without pre-resolving it.
535535

536+
If you are using a managed memcached service from [Google Cloud](https://cloud.google.com/memorystore/docs/memcached/auto-discovery-overview), or [AWS](https://docs.aws.amazon.com/AmazonElastiCache/latest/mem-ug/AutoDiscovery.HowAutoDiscoveryWorks.html), use the [auto-discovery](./config-file-reference.md#memcached-client-config) flag instead of DNS discovery, then use the discovery/configuration endpoint as the domain name without any prefix.
537+
536538
## Logging of IP of reverse proxy
537539

538540
If a reverse proxy is used in front of Cortex it might be diffult to troubleshoot errors. The following 3 settings can be used to log the IP address passed along by the reverse proxy in headers like X-Forwarded-For.

‎docs/configuration/config-file-reference.md

+15
Original file line numberDiff line numberDiff line change
@@ -4765,6 +4765,11 @@ bucket_store:
47654765
# CLI flag: -blocks-storage.bucket-store.index-cache.memcached.max-item-size
47664766
[max_item_size: <int> | default = 1048576]
47674767
4768+
# Use memcached auto-discovery mechanism provided by some cloud provider
4769+
# like GCP and AWS
4770+
# CLI flag: -blocks-storage.bucket-store.index-cache.memcached.auto-discovery
4771+
[auto_discovery: <boolean> | default = false]
4772+
47684773
chunks_cache:
47694774
# Backend for chunks cache, if not empty. Supported values: memcached.
47704775
# CLI flag: -blocks-storage.bucket-store.chunks-cache.backend
@@ -4812,6 +4817,11 @@ bucket_store:
48124817
# CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.max-item-size
48134818
[max_item_size: <int> | default = 1048576]
48144819
4820+
# Use memcached auto-discovery mechanism provided by some cloud provider
4821+
# like GCP and AWS
4822+
# CLI flag: -blocks-storage.bucket-store.chunks-cache.memcached.auto-discovery
4823+
[auto_discovery: <boolean> | default = false]
4824+
48154825
# Size of each subrange that bucket object is split into for better caching.
48164826
# CLI flag: -blocks-storage.bucket-store.chunks-cache.subrange-size
48174827
[subrange_size: <int> | default = 16000]
@@ -4877,6 +4887,11 @@ bucket_store:
48774887
# CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.max-item-size
48784888
[max_item_size: <int> | default = 1048576]
48794889
4890+
# Use memcached auto-discovery mechanism provided by some cloud provider
4891+
# like GCP and AWS
4892+
# CLI flag: -blocks-storage.bucket-store.metadata-cache.memcached.auto-discovery
4893+
[auto_discovery: <boolean> | default = false]
4894+
48804895
# How long to cache list of tenants in the bucket.
48814896
# CLI flag: -blocks-storage.bucket-store.metadata-cache.tenants-list-ttl
48824897
[tenants_list_ttl: <duration> | default = 15m]

‎go.mod

+1-1
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ require (
5050
github.com/sony/gobreaker v0.4.1
5151
github.com/spf13/afero v1.2.2
5252
github.com/stretchr/testify v1.7.0
53-
github.com/thanos-io/thanos v0.22.0
53+
github.com/thanos-io/thanos v0.19.1-0.20210803192524-baea4ce9ef52
5454
github.com/uber/jaeger-client-go v2.29.1+incompatible
5555
github.com/weaveworks/common v0.0.0-20210901124008-1fa3f9fa874c
5656
go.etcd.io/bbolt v1.3.6

‎go.sum

+3-2
Original file line numberDiff line numberDiff line change
@@ -1728,8 +1728,8 @@ github.com/thanos-io/thanos v0.13.1-0.20210204123931-82545cdd16fe/go.mod h1:ZLDG
17281728
github.com/thanos-io/thanos v0.13.1-0.20210224074000-659446cab117/go.mod h1:kdqFpzdkveIKpNNECVJd75RPvgsAifQgJymwCdfev1w=
17291729
github.com/thanos-io/thanos v0.13.1-0.20210226164558-03dace0a1aa1/go.mod h1:gMCy4oCteKTT7VuXVvXLTPGzzjovX1VPE5p+HgL1hyU=
17301730
github.com/thanos-io/thanos v0.13.1-0.20210401085038-d7dff0c84d17/go.mod h1:zU8KqE+6A+HksK4wiep8e/3UvCZLm+Wrw9AqZGaAm9k=
1731-
github.com/thanos-io/thanos v0.22.0 h1:bHTzC0ZaP5rBJ2pOeJ73+hO/p3U4tSshJoR3pumM6sA=
1732-
github.com/thanos-io/thanos v0.22.0/go.mod h1:SZDWz3phcUcBr4MYFoPFRvl+Z9Nbi45HlwQlwSZSt+Q=
1731+
github.com/thanos-io/thanos v0.19.1-0.20210803192524-baea4ce9ef52 h1:aRxV+ebdkd5XqY1+vkwmdOpc66OpWtcvSF1e0qiRKsU=
1732+
github.com/thanos-io/thanos v0.19.1-0.20210803192524-baea4ce9ef52/go.mod h1:Xskx78e0CYL6w0yDNOZHGdvwQMlsuzPsePmPtbp9Xuk=
17331733
github.com/themihai/gomemcache v0.0.0-20180902122335-24332e2d58ab h1:7ZR3hmisBWw77ZpO1/o86g+JV3VKlk3d48jopJxzTjU=
17341734
github.com/themihai/gomemcache v0.0.0-20180902122335-24332e2d58ab/go.mod h1:eheTFp954zcWZXCU8d0AT76ftsQOTo4DTqkN/h3k1MY=
17351735
github.com/tidwall/pretty v0.0.0-20180105212114-65a9db5fad51/go.mod h1:XNkn88O1ChpSDQmQeStsy+sBenx6DDtFZJxhVysOjyk=
@@ -1915,6 +1915,7 @@ go.uber.org/atomic v1.8.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
19151915
go.uber.org/atomic v1.9.0 h1:ECmE8Bn/WFTYwEW/bpKD3M8VtR/zQVbavAoalC1PYyE=
19161916
go.uber.org/atomic v1.9.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
19171917
go.uber.org/automaxprocs v1.2.0/go.mod h1:YfO3fm683kQpzETxlTGZhGIVmXAhaw3gxeBADbpZtnU=
1918+
go.uber.org/automaxprocs v1.4.0/go.mod h1:/mTEdr7LvHhs0v7mjdxDreTz1OG5zdZGqgOnhWiR/+Q=
19181919
go.uber.org/goleak v1.0.0/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
19191920
go.uber.org/goleak v1.1.10 h1:z+mqJhf6ss6BSfSM671tgKyZBFPTTJM+HLxnhPC3wu0=
19201921
go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=

‎pkg/storage/tsdb/memcache_client_config.go

+3
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ type MemcachedClientConfig struct {
1818
MaxGetMultiConcurrency int `yaml:"max_get_multi_concurrency"`
1919
MaxGetMultiBatchSize int `yaml:"max_get_multi_batch_size"`
2020
MaxItemSize int `yaml:"max_item_size"`
21+
AutoDiscovery bool `yaml:"auto_discovery"`
2122
}
2223

2324
func (cfg *MemcachedClientConfig) RegisterFlagsWithPrefix(f *flag.FlagSet, prefix string) {
@@ -29,6 +30,7 @@ func (cfg *MemcachedClientConfig) RegisterFlagsWithPrefix(f *flag.FlagSet, prefi
2930
f.IntVar(&cfg.MaxGetMultiConcurrency, prefix+"max-get-multi-concurrency", 100, "The maximum number of concurrent connections running get operations. If set to 0, concurrency is unlimited.")
3031
f.IntVar(&cfg.MaxGetMultiBatchSize, prefix+"max-get-multi-batch-size", 0, "The maximum number of keys a single underlying get operation should run. If more keys are specified, internally keys are split into multiple batches and fetched concurrently, honoring the max concurrency. If set to 0, the max batch size is unlimited.")
3132
f.IntVar(&cfg.MaxItemSize, prefix+"max-item-size", 1024*1024, "The maximum size of an item stored in memcached. Bigger items are not stored. If set to 0, no maximum size is enforced.")
33+
f.BoolVar(&cfg.AutoDiscovery, prefix+"auto-discovery", false, "Use memcached auto-discovery mechanism provided by some cloud provider like GCP and AWS")
3234
}
3335

3436
func (cfg *MemcachedClientConfig) GetAddresses() []string {
@@ -59,5 +61,6 @@ func (cfg MemcachedClientConfig) ToMemcachedClientConfig() cacheutil.MemcachedCl
5961
MaxGetMultiBatchSize: cfg.MaxGetMultiBatchSize,
6062
MaxItemSize: model.Bytes(cfg.MaxItemSize),
6163
DNSProviderUpdateInterval: 30 * time.Second,
64+
AutoDiscovery: cfg.AutoDiscovery,
6265
}
6366
}

‎vendor/github.com/thanos-io/thanos/pkg/block/metadata/meta.go

+1
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎vendor/github.com/thanos-io/thanos/pkg/cacheutil/memcached_client.go

+41-16
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)