Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix/concurrent map interaction #4716

Merged

Conversation

alanprot
Copy link
Member

@alanprot alanprot commented Apr 12, 2022

What this PR does:

Fix race condition on /series.

ForReplicationSet can still invoke the callback after the quorum has reached and this can cause the metrics map to be write and read at the same time.

=== RUN   TestDistributor_MetricsForLabelMatchers_SingleSlowIngester
fatal error: concurrent map iteration and map write

goroutine 89 [running]:
runtime.throw({0x1efcf2d, 0x0})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc000a47c28 sp=0xc000a47bf8 pc=0x10383b1
runtime.mapiternext(0xc000a47da0)
	/usr/local/go/src/runtime/map.go:858 +0x4eb fp=0xc000a47c98 sp=0xc000a47c28 pc=0x10112cb
github.com/cortexproject/cortex/pkg/distributor.(*Distributor).MetricsForLabelMatchers(0x0, {0x21c53d8, 0xc0007f13e0}, 0x1ec9e30, 0x4, {0xc000a47ee0, 0x1, 0x1})
	/Users/approtas/workspaces/cortex/pkg/distributor/distributor.go:985 +0x325 fp=0xc000a47e10 sp=0xc000a47c98 pc=0x1bef3e5
github.com/cortexproject/cortex/pkg/distributor.TestDistributor_MetricsForLabelMatchers_SingleSlowIngester(0x1)
	/Users/approtas/workspaces/cortex/pkg/distributor/distributor_test.go:1870 +0x30e fp=0xc000a47f70 sp=0xc000a47e10 pc=0x1c09b8e
testing.tRunner(0xc0001d0d00, 0x1f3d210)
	/usr/local/go/src/testing/testing.go:1259 +0x102 fp=0xc000a47fc0 sp=0xc000a47f70 pc=0x11156c2

Checklist

  • Tests updated
  • [ NA] Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@alanprot alanprot merged commit 6758f1f into cortexproject:master Apr 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants