Querier segfault

[nickmonad]

Describe the bug
Observed panics due to segmentation faults in the querier.

To Reproduce
Run cortex v1.10.0-rc1 and utilize the querier

Environment:

• Infrastructure: AWS Fargate
• Deployment tool: terraform

Storage Engine

• Blocks
• Chunks

Additional Context

 error panic: runtime error: invalid memory address or nil pointer dereference [recovered]
 error 	panic: runtime error: invalid memory address or nil pointer dereference
 [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1448fb5]
 goroutine 936 [running]:
 github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1(0xc000c263a8, 0x2c0b9a0, 0xc00141ed80)
 	/__w/cortex/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:150 +0x1ab
 panic(0x232d460, 0x3c58790)
 	/usr/local/go/src/runtime/panic.go:965 +0x1b9
 github.com/cortexproject/cortex/pkg/util/extract.MetricNameMatcherFromMatchers(0xc001b70578, 0x1, 0x1, 0xc001bac150, 0x8, 0x0, 0x0, 0x14f9345)
 	/__w/cortex/cortex/pkg/util/extract/extract.go:58 +0x75
 github.com/cortexproject/cortex/pkg/distributor.(*Distributor).GetIngestersForQuery(0xc0007a2a00, 0x2bdeaa0, 0xc000fb9aa0, 0xc001b70578, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
 	/__w/cortex/cortex/pkg/distributor/query.go:138 +0x133
 github.com/cortexproject/cortex/pkg/distributor.(*Distributor).QueryExemplars.func1(0x2bdeaa0, 0xc000fb9aa0, 0x1a, 0xc039243a0f1ddfae)
 	/__w/cortex/cortex/pkg/distributor/query.go:67 +0x119
 github.com/weaveworks/common/instrument.CollectedRequest(0x2bde9f8, 0xc001b39bc0, 0x2733bfe, 0x1a, 0x2bce420, 0xc00028cc90, 0x2892ee8, 0xc001b70790, 0x0, 0x0)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/instrument/instrument.go:152 +0x271
 github.com/cortexproject/cortex/pkg/distributor.(*Distributor).QueryExemplars(0xc0007a2a00, 0x2bde9f8, 0xc001b39bc0, 0x17af6b820a0, 0x17af801b7a0, 0xc000c264e0, 0x1, 0x1, 0xc0015d7e40, 0x20, ...)
 	/__w/cortex/cortex/pkg/distributor/query.go:60 +0x125
 github.com/cortexproject/cortex/pkg/querier.(*distributorExemplarQuerier).Select(0xc0015d7e40, 0x17af6b820a0, 0x17af801b7a0, 0xc000c264e0, 0x1, 0x1, 0x0, 0xed8961004, 0x0, 0x0, ...)
 	/__w/cortex/cortex/pkg/querier/distributor_queryable.go:228 +0x9e
 github.com/prometheus/prometheus/web/api/v1.(*API).queryExemplars(0xc000387200, 0xc000364400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
 	/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:509 +0x9cb
 github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x2bcd280, 0xc0015d7d40, 0xc000364400)
 	/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:264 +0xa5
 net/http.HandlerFunc.ServeHTTP(0xc00000c8a0, 0x2bcd280, 0xc0015d7d40, 0xc000364400)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x2b9b720, 0xc00000c8a0, 0x2bd2a10, 0xc0015d7d00, 0xc000364400)
 	/__w/cortex/cortex/vendor/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7e
 github.com/prometheus/common/route.(*Router).handle.func1(0x2bd2a10, 0xc0015d7d00, 0xc000364300, 0x0, 0x0, 0x0)
 	/__w/cortex/cortex/vendor/github.com/prometheus/common/route/route.go:83 +0x27f
 github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0007f2f60, 0x2bd2a10, 0xc0015d7d00, 0xc000364300)
 	/__w/cortex/cortex/vendor/github.com/julienschmidt/httprouter/router.go:387 +0xc7e
 github.com/prometheus/common/route.(*Router).ServeHTTP(0xc000c2b280, 0x2bd2a10, 0xc0015d7d00, 0xc000364300)
 	/__w/cortex/cortex/vendor/github.com/prometheus/common/route/route.go:121 +0x4c
 github.com/cortexproject/cortex/pkg/api.getHTTPCacheGenNumberHeaderSetterMiddleware.func1.1(0x2bd2a10, 0xc0015d7d00, 0xc000364300)
 	/__w/cortex/cortex/pkg/api/middlewares.go:26 +0x206
 net/http.HandlerFunc.ServeHTTP(0xc0015d7c20, 0x2bd2a10, 0xc0015d7d00, 0xc000364300)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/weaveworks/common/middleware.Instrument.Wrap.func1.2(0x2bd2a10, 0xc0015d7d00)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:68 +0x4c
 github.com/felixge/httpsnoop.CaptureMetricsFn(0x2bd2a10, 0xc0015d7c00, 0xc000a47300, 0x2, 0x2c07008, 0xc001b39b80)
 	/__w/cortex/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:81 +0x24b
 github.com/weaveworks/common/middleware.Instrument.Wrap.func1(0x2bd2a10, 0xc0015d7c00, 0xc000364300)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:67 +0x325
 net/http.HandlerFunc.ServeHTTP(0xc000c31540, 0x2bd2a10, 0xc0015d7c00, 0xc000364300)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/gorilla/mux.(*Router).ServeHTTP(0xc00031a840, 0x2bd2a10, 0xc0015d7c00, 0xc000364100)
 	/__w/cortex/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0xd3
 github.com/cortexproject/cortex/pkg/querier/stats.WallTimeMiddleware.Wrap.func1(0x2bd2a10, 0xc0015d7c00, 0xc000364100)
 	/__w/cortex/cortex/pkg/querier/stats/time_middleware.go:20 +0x1e7
 net/http.HandlerFunc.ServeHTTP(0xc00000d6e0, 0x2bd2a10, 0xc0015d7c00, 0xc000364100)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/weaveworks/common/middleware.glob..func1.1(0x2bd2a10, 0xc0015d7c00, 0xc000333e00)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x147
 net/http.HandlerFunc.ServeHTTP(0xc00000d788, 0x2bd2a10, 0xc0015d7c00, 0xc000333e00)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/gorilla/mux.(*Router).ServeHTTP(0xc000178c00, 0x2bd2a10, 0xc0015d7c00, 0xc000333700)
 	/__w/cortex/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0xd3
 github.com/weaveworks/common/middleware.Instrument.Wrap.func1.2(0x2bd2a10, 0xc0015d7c00)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:68 +0x4c
 github.com/felixge/httpsnoop.CaptureMetricsFn(0x2bce450, 0xc000c31400, 0xc000a47990, 0x2, 0x2c07008, 0xc001b39b40)
 	/__w/cortex/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:81 +0x24b
 github.com/weaveworks/common/middleware.Instrument.Wrap.func1(0x2bce450, 0xc000c31400, 0xc000333700)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:67 +0x325
 net/http.HandlerFunc.ServeHTTP(0xc00009b3b0, 0x2bce450, 0xc000c31400, 0xc000333700)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/weaveworks/common/middleware.Log.Wrap.func1(0x2bd2710, 0xc0015d7b40, 0xc000333700)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:52 +0x1a9
 net/http.HandlerFunc.ServeHTTP(0xc0006d1100, 0x2bd2710, 0xc0015d7b40, 0xc000333700)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5(0x2bd0be0, 0xc001b39a40, 0xc000333500)
 	/__w/cortex/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:154 +0x5e4
 net/http.HandlerFunc.ServeHTTP(0xc0006d1140, 0x2bd0be0, 0xc001b39a40, 0xc000333500)
 	/usr/local/go/src/net/http/server.go:2069 +0x44
 github.com/weaveworks/common/httpgrpc/server.Server.Handle(0x2b9b720, 0xc0006d1140, 0x2bde9f8, 0xc0002b9a40, 0xc001ba19a0, 0x1, 0x1, 0xc0008e8f38)
 	/__w/cortex/cortex/vendor/github.com/weaveworks/common/httpgrpc/server/server.go:61 +0x2f1
 github.com/cortexproject/cortex/pkg/querier/worker.(*frontendProcessor).runRequest(0xc000bbd180, 0x2bde9f8, 0xc0002b9a40, 0xc001ba19a0, 0x0, 0xc001ba2780)
 	/__w/cortex/cortex/pkg/querier/worker/frontend_processor.go:123 +0x89
 created by github.com/cortexproject/cortex/pkg/querier/worker.(*frontendProcessor).process
 	/__w/cortex/cortex/pkg/querier/worker/frontend_processor.go:98 +0x16a

[bradleypettit]

I had the same behaviour upgrading our Cortex environment from v1.9.0 to v1.10.0. My dev environment is running 1 instance of each microservice on a single EC2 instance (CentOS 7).

We're running block storage (S3) with the following services: alertmanager, compactor, distributor, ingester, querier, ruler, store-gateway.

After upgrading all services to v1.10.0 we started seeing the below panic from the Querier service only.

2021/08/07 12:17:32 http: panic serving <ip_address>:37696: runtime error: invalid memory address or nil pointer dereference
goroutine 2231 [running]:
net/http.(*conn).serve.func1(0xc000511040)
        /usr/local/go/src/net/http/server.go:1824 +0x153
panic(0x232d460, 0x3c6e7b0)
        /usr/local/go/src/runtime/panic.go:971 +0x499
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5.1(0xc0007ca600, 0x2c0d800, 0xc000770c60)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:150 +0x1ab
panic(0x232d460, 0x3c6e7b0)
        /usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/cortexproject/cortex/pkg/util/extract.MetricNameMatcherFromMatchers(0xc000348378, 0x1, 0x1, 0xc0006ee4a0, 0x8, 0x0, 0x0, 0x14f9345)
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/util/extract/extract.go:58 +0x75
github.com/cortexproject/cortex/pkg/distributor.(*Distributor).GetIngestersForQuery(0xc000148000, 0x2be0900, 0xc00020e120, 0xc000348378, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...)
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/distributor/query.go:138 +0x133
github.com/cortexproject/cortex/pkg/distributor.(*Distributor).QueryExemplars.func1(0x2be0900, 0xc00020e120, 0x1a, 0xc03b992f0806a008)
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/distributor/query.go:67 +0x119
github.com/weaveworks/common/instrument.CollectedRequest(0x2be0858, 0xc00037ce00, 0x2733bfe, 0x1a, 0x2bd0280, 0xc000121098, 0x2894d30, 0xc000348590, 0x10, 0xc001afe380)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/instrument/instrument.go:152 +0x271
github.com/cortexproject/cortex/pkg/distributor.(*Distributor).QueryExemplars(0xc000148000, 0x2be0858, 0xc00037ce00, 0x17b1e6216b0, 0x17b1e66aa90, 0xc0007ca768, 0x1, 0x1, 0xc00088c540, 0x20, ...)
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/distributor/query.go:60 +0x125
github.com/cortexproject/cortex/pkg/querier.(*distributorExemplarQuerier).Select(0xc00088c540, 0x17b1e6216b0, 0x17b1e66aa90, 0xc0007ca768, 0x1, 0x1, 0x0, 0xed89fe43a, 0x0, 0x0, ...)
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/querier/distributor_queryable.go:228 +0x9e
github.com/prometheus/prometheus/web/api/v1.(*API).queryExemplars(0xc00095e6c0, 0xc00013fd00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:509 +0x9cb
github.com/prometheus/prometheus/web/api/v1.(*API).Register.func1.1(0x2bcf0e0, 0xc00088c3e0, 0xc00013fd00)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/web/api/v1/api.go:264 +0xa5
net/http.HandlerFunc.ServeHTTP(0xc0001178a8, 0x2bcf0e0, 0xc00088c3e0, 0xc00013fd00)
        /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/prometheus/prometheus/util/httputil.CompressionHandler.ServeHTTP(0x2b9d580, 0xc0001178a8, 0x2bd4870, 0xc00088c380, 0xc00013fd00)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/prometheus/util/httputil/compression.go:90 +0x7e
github.com/prometheus/common/route.(*Router).handle.func1(0x2bd4870, 0xc00088c380, 0xc00013fb00, 0x0, 0x0, 0x0)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route/route.go:83 +0x27f
github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc0002e7c80, 0x2bd4870, 0xc00088c380, 0xc00013fb00)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/julienschmidt/httprouter/router.go:387 +0xc7e
github.com/prometheus/common/route.(*Router).ServeHTTP(0xc000a3c740, 0x2bd4870, 0xc00088c380, 0xc00013fb00)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/prometheus/common/route/route.go:121 +0x4c
github.com/cortexproject/cortex/pkg/api.getHTTPCacheGenNumberHeaderSetterMiddleware.func1.1(0x2bd4870, 0xc00088c380, 0xc00013fb00)
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/api/middlewares.go:26 +0x206
net/http.HandlerFunc.ServeHTTP(0xc00088c2a0, 0x2bd4870, 0xc00088c380, 0xc00013fb00)
        /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/weaveworks/common/middleware.Instrument.Wrap.func1.2(0x2bd4870, 0xc00088c380)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:68 +0x4c
github.com/felixge/httpsnoop.CaptureMetricsFn(0x2bd4870, 0xc00088c280, 0xc000ad9100, 0x2, 0x2c08e68, 0xc000e6afc0)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:81 +0x24b
github.com/weaveworks/common/middleware.Instrument.Wrap.func1(0x2bd4870, 0xc00088c280, 0xc00013fb00)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:67 +0x325
net/http.HandlerFunc.ServeHTTP(0xc0011a8d70, 0x2bd4870, 0xc00088c280, 0xc00013fb00)
        /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000a4a0c0, 0x2bd4870, 0xc00088c280, 0xc00013f900)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0xd3
github.com/cortexproject/cortex/pkg/querier/stats.WallTimeMiddleware.Wrap.func1(0x2bd4870, 0xc00088c280, 0xc00013f900)
        /home/vagrant/src/github.com/cortexproject/cortex/pkg/querier/stats/time_middleware.go:20 +0x1e7
net/http.HandlerFunc.ServeHTTP(0xc0009144e0, 0x2bd4870, 0xc00088c280, 0xc00013f900)
        /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/weaveworks/common/middleware.glob..func1.1(0x2bd4870, 0xc00088c280, 0xc00013ea00)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x147
net/http.HandlerFunc.ServeHTTP(0xc0009145a0, 0x2bd4870, 0xc00088c280, 0xc00013ea00)
        /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0006a06c0, 0x2bd4870, 0xc00088c280, 0xc00013e800)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/gorilla/mux/mux.go:212 +0xd3
github.com/weaveworks/common/middleware.Instrument.Wrap.func1.2(0x2bd4870, 0xc00088c280)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:68 +0x4c
github.com/felixge/httpsnoop.CaptureMetricsFn(0x2bd02b0, 0xc0011a8cd0, 0xc000ad9790, 0x2, 0x2c08e68, 0xc000e6af80)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/felixge/httpsnoop/capture_metrics.go:81 +0x24b
github.com/weaveworks/common/middleware.Instrument.Wrap.func1(0x2bd02b0, 0xc0011a8cd0, 0xc00013e800)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:67 +0x325
net/http.HandlerFunc.ServeHTTP(0xc0009269b0, 0x2bd02b0, 0xc0011a8cd0, 0xc00013e800)
        /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/weaveworks/common/middleware.Log.Wrap.func1(0x2bd4990, 0xc0011a8c80, 0xc00013e800)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:52 +0x1a9
net/http.HandlerFunc.ServeHTTP(0xc000a26240, 0x2bd4990, 0xc0011a8c80, 0xc00013e800)
        /usr/local/go/src/net/http/server.go:2069 +0x44
github.com/opentracing-contrib/go-stdlib/nethttp.MiddlewareFunc.func5(0x2bd29e0, 0xc00066e000, 0xc00013e700)
        /home/vagrant/src/github.com/cortexproject/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:154 +0x5e4
net/http.HandlerFunc.ServeHTTP(0xc000a26280, 0x2bd29e0, 0xc00066e000, 0xc00013e700)
        /usr/local/go/src/net/http/server.go:2069 +0x44
net/http.serverHandler.ServeHTTP(0xc000648460, 0x2bd29e0, 0xc00066e000, 0xc00013e700)
        /usr/local/go/src/net/http/server.go:2887 +0xa3
net/http.(*conn).serve(0xc000511040, 0x2be0900, 0xc00037c8c0)
        /usr/local/go/src/net/http/server.go:1952 +0x8cd
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:3013 +0x39b

[blind-oracle]

Same for me on release 1.10

[RicardoNalesAmato]

Same for us after upgrading from 1.9 to 1.10 using block storage and only one tenant.

[blind-oracle]

@RicardoNalesAmato I've worked this around by replacing a function that serves Exemplar with a stub that returns nothing as we don't need them.

[pstibrany]

I believe this is fixed by #4473.

[pstibrany]

Do you happen to use -distributor.shard-by-all-labels=false? Setting it to true is a workaround as well, but it will cause reshuffling of series across ingesters.

[nickmonad]

@pstibrany We will try that fix in our staging cluster. Thanks!

[nickmonad]

I can confirm that deploying a version of cortex that includes #4473 has fixed the issue for us

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.