ingest-storage: Add consumer support for a new record version 2, based on Remote Write 2.0 #11406
Conversation
| }) | ||
| } | ||
|
|
||
| func BenchmarkDeserializeRecordContent(b *testing.B) { |
There was a problem hiding this comment.
For those curious, the v2 benchmark gives one extra allocation per timeseries over v1. This PR doesn't try to add any optimizations beyond what's already there. This is on par with my initial test results.
go test ./pkg/storage/ingest/... -run ^$ -bench BenchmarkDeserializeRecordContent -benchtime=1000x -benchmem
BenchmarkDeserializeRecordContent/deserialize_v1-16 1000 5242524 ns/op 12498059 B/op 39975 allocs/op
BenchmarkDeserializeRecordContent/deserialize_v2-16 1000 4888400 ns/op 12886412 B/op 49978 allocs/op
There was a problem hiding this comment.
Please use benchstat next time to compare benchmarks 😉 Run each benchmark at least 6 times to get better results from benchstat.
pkg/storage/ingest/pusher.go
Outdated
| @@ -66,7 +66,7 @@ func (c pusherConsumer) Consume(ctx context.Context, records []record) (returnEr | |||
| }(time.Now()) | |||
|
|
|||
| type parsedRecord struct { | |||
| *mimirpb.WriteRequest | |||
| *mimirpb.PreallocWriteRequest | |||
There was a problem hiding this comment.
This PR depends on #11393. I inlined those changes into this PR until that's merged.
This is logically equivalent to:
WriteRequest: &mimirpb.WriteRequest{
Timeseries: mimirpb.PreallocTimeseriesSliceFromPool(),
},
PreallocWriteRequest.Unmarshal() calls PreallocTimeseriesSliceFromPool()
alexweav
changed the title
| }) | ||
| } | ||
|
|
||
| func BenchmarkDeserializeRecordContent(b *testing.B) { |
There was a problem hiding this comment.
Please use benchstat next time to compare benchmarks 😉 Run each benchmark at least 6 times to get better results from benchstat.
alexweav
force-pushed
the
alexweav/rw-v2-consumer
branch
from
c28c1f3 to
f24b2e6
Compare
alexweav
force-pushed
the
alexweav/rw-v2-consumer
branch
from
f24b2e6 to
d638be4
Compare
What this PR does
Introduces a new record version
V2, and implements consumer-side support for it.It's inspired by Prometheus Remote Write 2.0, but not strictly compatible with it. A notable difference is that the symbol refs are shifted up by 64. This reserves space for a future common symbols table, allowing us to completely avoid sending certain commonly used symbols through Kafka.
The format is internal to Mimir, it doesn't need to exactly match any external standard. Aside from within test code, there is currently no way to produce records of the V2 format. Therefore, we're still able to iterate on the format in the future and make breaking changes. I'm not adding this to the changelog for that reason; when we decide to "lock down" the format, the changelog will be updated at that time.
The implementation heavily leverages @krajorama's work in #11100 and inherits several optimizations from it.
Which issue(s) this PR fixes or relates to
Rel https://github.com/grafana/mimir-squad/issues/2253
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.