Hook up partition compaction end to end implementation #6510

alexqyle · 2025-01-15T22:13:18Z

What this PR does:

Implement partitioning compaction related lifecycle functions to make partitioning compaction end to end working.

PartitionCompactionBlockDeletableChecker makes sure no parent blocks got deleted after each compaction. Cleaner would handle parent blocks clean up for partitioning compaction.
ShardedBlockPopulator would use ShardedPosting to including particular series in the result block.
ShardedCompactionLifecycleCallbackis used to emit partitioning compaction metrics at beginning and end of compaction. It also initialize ShardedBlockPopulator for each compaction.

Which issue(s) this PR fixes:
Fixes #

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

danielblando

LGTM

yeya24

LGTM

yeya24 · 2025-01-17T00:07:17Z

pkg/compactor/compactor.go

@@ -698,15 +754,26 @@ func (c *Compactor) stopping(_ error) error {
 }

 func (c *Compactor) running(ctx context.Context) error {
+	// Ensure an initial cleanup occurred as first thing when running compactor.
+	if err := services.StartAndAwaitRunning(ctx, c.blocksCleaner); err != nil {


Is there a specific reason why we have to move cleaning here?

Because cleaner cycle might be running for a while depending on how many tenants and how big each tenants are. We don't want to compactor got into unhealthy state in the ring because of long running cleaner process.

yeya24 · 2025-01-17T00:09:40Z

pkg/compactor/compactor.go

+
+func (f *DisabledDeduplicateFilter) DuplicateIDs() []ulid.ULID {
+	return nil
+}


Can we make these types private?
Can you add some comment to DisabledDeduplicateFilter. We want to disable duplicate filter because it makes no sense for partitioning compactor as we always have duplicates?

The DefaultDeduplicateFilter from Thanos would mark duplicate blocks if those blocks are from same group and having same source blocks. In partitioning compactor, partitions from same time range would always or eventually have same source as it is the natural of partitioning compactor. We don't want those blocks got filtered out when doing grouping for next level compaction.

yeya24 · 2025-01-17T00:10:38Z

pkg/compactor/sharded_block_populator.go

+
+	globalMaxt := blocks[0].Meta().MaxTime
+	g, _ := errgroup.WithContext(ctx)
+	g.SetLimit(8)


Is this a sane default to set in Cortex?

With my test 8 is good enough to keep CPU busy during compaction. I am wondering if this number is too high for end user, would this just cause CPU usage peaked at 100% for longer time?

pkg/compactor/sharded_block_populator.go

pkg/compactor/sharded_posting.go

pkg/compactor/sharded_block_populator.go

…fecycle Signed-off-by: Alex Le <[email protected]>

Signed-off-by: Alex Le <[email protected]>

…#6510) * Implemented partition compaction end to end with custom compaction lifecycle Signed-off-by: Alex Le <[email protected]> * removed unused variable Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * refactor according to comments Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * check context error inside sharded posting Signed-off-by: Alex Le <[email protected]> * fix lint Signed-off-by: Alex Le <[email protected]> * fix integration test for memberlist Signed-off-by: Alex Le <[email protected]> * make compactor initial wait cancellable Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]>

* Purge expired postings cache items due inactivity (#6502) * Purge expired postings cache items due inactivity Signed-off-by: alanprot <[email protected]> * Fix comments Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Update thanos to 4ba0ba403896 (#6503) * Update thanos to 4ba0ba403896 Signed-off-by: Daniel Sabsay <[email protected]> * run go mod vendor Signed-off-by: Daniel Sabsay <[email protected]> --------- Signed-off-by: Daniel Sabsay <[email protected]> Co-authored-by: Daniel Sabsay <[email protected]> Signed-off-by: Alex Le <[email protected]> * Bump the actions-dependencies group across 1 directory with 2 updates (#6505) Bumps the actions-dependencies group with 2 updates in the / directory: [actions/upload-artifact](https://github.com/actions/upload-artifact) and [github/codeql-action](https://github.com/github/codeql-action). Updates `actions/upload-artifact` from 4.5.0 to 4.6.0 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@6f51ac0...65c4c4a) Updates `github/codeql-action` from 3.28.0 to 3.28.1 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@48ab28a...b6a472f) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <[email protected]> * calculate # of concurrency only once at the runner (#6506) Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Implement partition compaction planner (#6469) * Implement partition compaction grouper Signed-off-by: Alex Le <[email protected]> * fix comment Signed-off-by: Alex Le <[email protected]> * replace level 1 compaction limits with ingestion replication factor Signed-off-by: Alex Le <[email protected]> * fix doc Signed-off-by: Alex Le <[email protected]> * update compaction_visit_marker_timeout default value Signed-off-by: Alex Le <[email protected]> * update default value for compactor_partition_index_size_limit_in_bytes Signed-off-by: Alex Le <[email protected]> * refactor code Signed-off-by: Alex Le <[email protected]> * address comments and refactor Signed-off-by: Alex Le <[email protected]> * address comment Signed-off-by: Alex Le <[email protected]> * address comment Signed-off-by: Alex Le <[email protected]> * update config name Signed-off-by: Alex Le <[email protected]> * Implement partition compaction planner Signed-off-by: Alex Le <[email protected]> * fix after rebase Signed-off-by: Alex Le <[email protected]> * addressed comments Signed-off-by: Alex Le <[email protected]> * updated doc and refactored metric Signed-off-by: Alex Le <[email protected]> * fix test Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Add max tenant config to tenant federation (#6493) Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Add cleaner logic to clean partition compaction blocks and related files (#6507) * Add cleaner logic to clean partition compaction blocks and related files Signed-off-by: Alex Le <[email protected]> * refactored metrics Signed-off-by: Alex Le <[email protected]> * refactor Signed-off-by: Alex Le <[email protected]> * update logs Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Update RELEASE.md (#6511) Maintainers would like an additional week to get the partition compactor changes in before the first release candidate for 1.19. Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * update thanos version to 236777732278c64ca01c1c09d726f0f712c87164 (#6514) Signed-off-by: yeya24 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Fix race that can cause nil reference when using expanded postings (#6518) Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Add more op label values to cortex_query_frontend_queries_total metric (#6519) Signed-off-by: Alex Le <[email protected]> * Allow use of non-dualstack endpoints for S3 blocks storage (#6522) Signed-off-by: Alex Le <[email protected]> * Expose grpc client connect timeout config and default to 5s (#6523) * expose grpc client connect timeout config Signed-off-by: yeya24 <[email protected]> * changelog Signed-off-by: yeya24 <[email protected]> --------- Signed-off-by: yeya24 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Hook up partition compaction end to end implementation (#6510) * Implemented partition compaction end to end with custom compaction lifecycle Signed-off-by: Alex Le <[email protected]> * removed unused variable Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * refactor according to comments Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * check context error inside sharded posting Signed-off-by: Alex Le <[email protected]> * fix lint Signed-off-by: Alex Le <[email protected]> * fix integration test for memberlist Signed-off-by: Alex Le <[email protected]> * make compactor initial wait cancellable Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Test for nil on expire expanded postings (#6521) * Test for nil on expire expanded postings Signed-off-by: alanprot <[email protected]> * stopping ingester Signed-off-by: alanprot <[email protected]> * refactor the test to not timeout Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * log when a request starts running in querier (#6525) * log when a request starts running in querier Signed-off-by: Ahmed Hassan <[email protected]> * log when a request starts running in querier for frontend processor Signed-off-by: Ahmed Hassan <[email protected]> --------- Signed-off-by: Ahmed Hassan <[email protected]> Signed-off-by: Alex Le <[email protected]> * Update build image according to 03a8f8c (#6508) Signed-off-by: Friedrich Gonzalez <[email protected]> Signed-off-by: Alex Le <[email protected]> * Deprecate -blocks-storage.tsdb.wal-compression-enabled flag Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Fix test (#6537) Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Alex Le <[email protected]> * Mark 1.19 release in progress https://github.com/cortexproject/cortex/blob/master/RELEASE.md#show-that-a-release-is-in-progress Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * Prepare 1.19.0-rc.0 Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * Revert "Prepare 1.19.0-rc.0" Signed-off-by: Alex Le <[email protected]> * Fixed blocksGroupWithPartition unable to reuse functions from blocksGroup (#6547) * Fixed blocksGroupWithPartition unable to reuse functions from blocksGroup Signed-off-by: Alex Le <[email protected]> * update tests Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Remove TransferChunks gRPC method (#6543) Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Uupdate Ppromqlsmith (#6557) Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Query Partial Data (#6526) * Create partial_data Signed-off-by: Justin Jung <[email protected]> * Fix lazyquery so that warning message is returned Signed-off-by: Justin Jung <[email protected]> * Add QueryPartialData limit Signed-off-by: Justin Jung <[email protected]> * Fix broken mock Signed-off-by: Justin Jung <[email protected]> * Make response with warnings to be not cached Signed-off-by: Justin Jung <[email protected]> * Updated streamingSelect in distributor_queryable Signed-off-by: Justin Jung <[email protected]> * Update query.go Signed-off-by: Justin Jung <[email protected]> * Update replication_set Signed-off-by: Justin Jung <[email protected]> * Lint Signed-off-by: Justin Jung <[email protected]> * Lint again Signed-off-by: Justin Jung <[email protected]> * Generated doc Signed-off-by: Justin Jung <[email protected]> * Changelog Signed-off-by: Justin Jung <[email protected]> * Update config description Signed-off-by: Justin Jung <[email protected]> * Do not remove warnings from seriesSet Signed-off-by: Justin Jung <[email protected]> * Avoid cache only if the warning message contains partial data error Signed-off-by: Justin Jung <[email protected]> * Remove context usage for partial data Signed-off-by: Justin Jung <[email protected]> * Refactor how partial data info is passed + apply to series and label methods as well Signed-off-by: Justin Jung <[email protected]> * Lint + fix tests Signed-off-by: Justin Jung <[email protected]> * Fix build Signed-off-by: Justin Jung <[email protected]> * Create separate config for ruler partial data Signed-off-by: Justin Jung <[email protected]> * Genereta doc Signed-off-by: Justin Jung <[email protected]> * Add more tests Signed-off-by: Justin Jung <[email protected]> * Change error Signed-off-by: Justin Jung <[email protected]> * Fix test Signed-off-by: Justin Jung <[email protected]> * Update changelog Signed-off-by: Justin Jung <[email protected]> * Update changelog Signed-off-by: Justin Jung <[email protected]> * Nit Signed-off-by: Justin Jung <[email protected]> * Nit Signed-off-by: Justin Jung <[email protected]> --------- Signed-off-by: Justin Jung <[email protected]> Signed-off-by: Alex Le <[email protected]> * Add timeout for dynamodb ring kv (#6544) * add dynamodb kv with timeout enforced Signed-off-by: yeya24 <[email protected]> * add tests Signed-off-by: yeya24 <[email protected]> * docs Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: yeya24 <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Alex Le <[email protected]> * Bump the actions-dependencies group across 1 directory with 2 updates (#6564) Bumps the actions-dependencies group with 2 updates in the / directory: [github/codeql-action](https://github.com/github/codeql-action) and [actions/setup-go](https://github.com/actions/setup-go). Updates `github/codeql-action` from 3.28.1 to 3.28.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@b6a472f...6e54559) Updates `actions/setup-go` from 5.2.0 to 5.3.0 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@3041bf5...f111f33) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <[email protected]> * Fix: expanded postings can cache wrong data when queries are issued "in the future" (#6562) * improve fuzz test for expanded postings cache Signed-off-by: alanprot <[email protected]> * create more tests on the expanded postings cache Signed-off-by: alanprot <[email protected]> * adding get series call on the test Signed-off-by: alanprot <[email protected]> * no use CachedBlockChunkQuerier when query time range is completely after the last sample added in the head Signed-off-by: alanprot <[email protected]> * adding comments Signed-off-by: alanprot <[email protected]> * increase the number of fuzz test from 100 to 300 Signed-off-by: alanprot <[email protected]> * add get series fuzzy testing Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Extend ShuffleSharding on READONLY ingesters (#6517) * Filter readOnly ingesters when sharding Signed-off-by: Daniel Deluiggi <[email protected]> * Extend shard on READONLY Signed-off-by: Daniel Deluiggi <[email protected]> * Remove old code Signed-off-by: Daniel Deluiggi <[email protected]> * Fix test Signed-off-by: Daniel Deluiggi <[email protected]> * update changelog Signed-off-by: Daniel Deluiggi <[email protected]> --------- Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Alex Le <[email protected]> * Create guide doc for partition compaction Signed-off-by: Alex Le <[email protected]> * Update docs/guides/partitioning-compactor.md Co-authored-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * updated doc Signed-off-by: Alex Le <[email protected]> * clean white space Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> Signed-off-by: Daniel Sabsay <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Charlie Le <[email protected]> Signed-off-by: yeya24 <[email protected]> Signed-off-by: Ahmed Hassan <[email protected]> Signed-off-by: Friedrich Gonzalez <[email protected]> Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Justin Jung <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Alex Le <[email protected]> Co-authored-by: Alan Protasio <[email protected]> Co-authored-by: Daniel Sabsay <[email protected]> Co-authored-by: Daniel Sabsay <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: SungJin1212 <[email protected]> Co-authored-by: Charlie Le <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Sam McBroom <[email protected]> Co-authored-by: Ahmed Hassan <[email protected]> Co-authored-by: Friedrich Gonzalez <[email protected]> Co-authored-by: Daniel Blando <[email protected]> Co-authored-by: Justin Jung <[email protected]>

* Purge expired postings cache items due inactivity (cortexproject#6502) * Purge expired postings cache items due inactivity Signed-off-by: alanprot <[email protected]> * Fix comments Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Update thanos to 4ba0ba403896 (cortexproject#6503) * Update thanos to 4ba0ba403896 Signed-off-by: Daniel Sabsay <[email protected]> * run go mod vendor Signed-off-by: Daniel Sabsay <[email protected]> --------- Signed-off-by: Daniel Sabsay <[email protected]> Co-authored-by: Daniel Sabsay <[email protected]> Signed-off-by: Alex Le <[email protected]> * Bump the actions-dependencies group across 1 directory with 2 updates (cortexproject#6505) Bumps the actions-dependencies group with 2 updates in the / directory: [actions/upload-artifact](https://github.com/actions/upload-artifact) and [github/codeql-action](https://github.com/github/codeql-action). Updates `actions/upload-artifact` from 4.5.0 to 4.6.0 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@6f51ac0...65c4c4a) Updates `github/codeql-action` from 3.28.0 to 3.28.1 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@48ab28a...b6a472f) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <[email protected]> * calculate # of concurrency only once at the runner (cortexproject#6506) Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Implement partition compaction planner (cortexproject#6469) * Implement partition compaction grouper Signed-off-by: Alex Le <[email protected]> * fix comment Signed-off-by: Alex Le <[email protected]> * replace level 1 compaction limits with ingestion replication factor Signed-off-by: Alex Le <[email protected]> * fix doc Signed-off-by: Alex Le <[email protected]> * update compaction_visit_marker_timeout default value Signed-off-by: Alex Le <[email protected]> * update default value for compactor_partition_index_size_limit_in_bytes Signed-off-by: Alex Le <[email protected]> * refactor code Signed-off-by: Alex Le <[email protected]> * address comments and refactor Signed-off-by: Alex Le <[email protected]> * address comment Signed-off-by: Alex Le <[email protected]> * address comment Signed-off-by: Alex Le <[email protected]> * update config name Signed-off-by: Alex Le <[email protected]> * Implement partition compaction planner Signed-off-by: Alex Le <[email protected]> * fix after rebase Signed-off-by: Alex Le <[email protected]> * addressed comments Signed-off-by: Alex Le <[email protected]> * updated doc and refactored metric Signed-off-by: Alex Le <[email protected]> * fix test Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Add max tenant config to tenant federation (cortexproject#6493) Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Add cleaner logic to clean partition compaction blocks and related files (cortexproject#6507) * Add cleaner logic to clean partition compaction blocks and related files Signed-off-by: Alex Le <[email protected]> * refactored metrics Signed-off-by: Alex Le <[email protected]> * refactor Signed-off-by: Alex Le <[email protected]> * update logs Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Update RELEASE.md (cortexproject#6511) Maintainers would like an additional week to get the partition compactor changes in before the first release candidate for 1.19. Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * update thanos version to 236777732278c64ca01c1c09d726f0f712c87164 (cortexproject#6514) Signed-off-by: yeya24 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Fix race that can cause nil reference when using expanded postings (cortexproject#6518) Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Add more op label values to cortex_query_frontend_queries_total metric (cortexproject#6519) Signed-off-by: Alex Le <[email protected]> * Allow use of non-dualstack endpoints for S3 blocks storage (cortexproject#6522) Signed-off-by: Alex Le <[email protected]> * Expose grpc client connect timeout config and default to 5s (cortexproject#6523) * expose grpc client connect timeout config Signed-off-by: yeya24 <[email protected]> * changelog Signed-off-by: yeya24 <[email protected]> --------- Signed-off-by: yeya24 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Hook up partition compaction end to end implementation (cortexproject#6510) * Implemented partition compaction end to end with custom compaction lifecycle Signed-off-by: Alex Le <[email protected]> * removed unused variable Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * refactor according to comments Signed-off-by: Alex Le <[email protected]> * tweak test Signed-off-by: Alex Le <[email protected]> * check context error inside sharded posting Signed-off-by: Alex Le <[email protected]> * fix lint Signed-off-by: Alex Le <[email protected]> * fix integration test for memberlist Signed-off-by: Alex Le <[email protected]> * make compactor initial wait cancellable Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Test for nil on expire expanded postings (cortexproject#6521) * Test for nil on expire expanded postings Signed-off-by: alanprot <[email protected]> * stopping ingester Signed-off-by: alanprot <[email protected]> * refactor the test to not timeout Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * log when a request starts running in querier (cortexproject#6525) * log when a request starts running in querier Signed-off-by: Ahmed Hassan <[email protected]> * log when a request starts running in querier for frontend processor Signed-off-by: Ahmed Hassan <[email protected]> --------- Signed-off-by: Ahmed Hassan <[email protected]> Signed-off-by: Alex Le <[email protected]> * Update build image according to cortexproject@03a8f8c (cortexproject#6508) Signed-off-by: Friedrich Gonzalez <[email protected]> Signed-off-by: Alex Le <[email protected]> * Deprecate -blocks-storage.tsdb.wal-compression-enabled flag Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Fix test (cortexproject#6537) Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Alex Le <[email protected]> * Mark 1.19 release in progress https://github.com/cortexproject/cortex/blob/master/RELEASE.md#show-that-a-release-is-in-progress Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * Prepare 1.19.0-rc.0 Signed-off-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * Revert "Prepare 1.19.0-rc.0" Signed-off-by: Alex Le <[email protected]> * Fixed blocksGroupWithPartition unable to reuse functions from blocksGroup (cortexproject#6547) * Fixed blocksGroupWithPartition unable to reuse functions from blocksGroup Signed-off-by: Alex Le <[email protected]> * update tests Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: Alex Le <[email protected]> * Remove TransferChunks gRPC method (cortexproject#6543) Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Alex Le <[email protected]> * Uupdate Ppromqlsmith (cortexproject#6557) Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Query Partial Data (cortexproject#6526) * Create partial_data Signed-off-by: Justin Jung <[email protected]> * Fix lazyquery so that warning message is returned Signed-off-by: Justin Jung <[email protected]> * Add QueryPartialData limit Signed-off-by: Justin Jung <[email protected]> * Fix broken mock Signed-off-by: Justin Jung <[email protected]> * Make response with warnings to be not cached Signed-off-by: Justin Jung <[email protected]> * Updated streamingSelect in distributor_queryable Signed-off-by: Justin Jung <[email protected]> * Update query.go Signed-off-by: Justin Jung <[email protected]> * Update replication_set Signed-off-by: Justin Jung <[email protected]> * Lint Signed-off-by: Justin Jung <[email protected]> * Lint again Signed-off-by: Justin Jung <[email protected]> * Generated doc Signed-off-by: Justin Jung <[email protected]> * Changelog Signed-off-by: Justin Jung <[email protected]> * Update config description Signed-off-by: Justin Jung <[email protected]> * Do not remove warnings from seriesSet Signed-off-by: Justin Jung <[email protected]> * Avoid cache only if the warning message contains partial data error Signed-off-by: Justin Jung <[email protected]> * Remove context usage for partial data Signed-off-by: Justin Jung <[email protected]> * Refactor how partial data info is passed + apply to series and label methods as well Signed-off-by: Justin Jung <[email protected]> * Lint + fix tests Signed-off-by: Justin Jung <[email protected]> * Fix build Signed-off-by: Justin Jung <[email protected]> * Create separate config for ruler partial data Signed-off-by: Justin Jung <[email protected]> * Genereta doc Signed-off-by: Justin Jung <[email protected]> * Add more tests Signed-off-by: Justin Jung <[email protected]> * Change error Signed-off-by: Justin Jung <[email protected]> * Fix test Signed-off-by: Justin Jung <[email protected]> * Update changelog Signed-off-by: Justin Jung <[email protected]> * Update changelog Signed-off-by: Justin Jung <[email protected]> * Nit Signed-off-by: Justin Jung <[email protected]> * Nit Signed-off-by: Justin Jung <[email protected]> --------- Signed-off-by: Justin Jung <[email protected]> Signed-off-by: Alex Le <[email protected]> * Add timeout for dynamodb ring kv (cortexproject#6544) * add dynamodb kv with timeout enforced Signed-off-by: yeya24 <[email protected]> * add tests Signed-off-by: yeya24 <[email protected]> * docs Signed-off-by: Ben Ye <[email protected]> * update changelog Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: yeya24 <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Alex Le <[email protected]> * Bump the actions-dependencies group across 1 directory with 2 updates (cortexproject#6564) Bumps the actions-dependencies group with 2 updates in the / directory: [github/codeql-action](https://github.com/github/codeql-action) and [actions/setup-go](https://github.com/actions/setup-go). Updates `github/codeql-action` from 3.28.1 to 3.28.7 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@b6a472f...6e54559) Updates `actions/setup-go` from 5.2.0 to 5.3.0 - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](actions/setup-go@3041bf5...f111f33) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions-dependencies - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions-dependencies ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Alex Le <[email protected]> * Fix: expanded postings can cache wrong data when queries are issued "in the future" (cortexproject#6562) * improve fuzz test for expanded postings cache Signed-off-by: alanprot <[email protected]> * create more tests on the expanded postings cache Signed-off-by: alanprot <[email protected]> * adding get series call on the test Signed-off-by: alanprot <[email protected]> * no use CachedBlockChunkQuerier when query time range is completely after the last sample added in the head Signed-off-by: alanprot <[email protected]> * adding comments Signed-off-by: alanprot <[email protected]> * increase the number of fuzz test from 100 to 300 Signed-off-by: alanprot <[email protected]> * add get series fuzzy testing Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> * Extend ShuffleSharding on READONLY ingesters (cortexproject#6517) * Filter readOnly ingesters when sharding Signed-off-by: Daniel Deluiggi <[email protected]> * Extend shard on READONLY Signed-off-by: Daniel Deluiggi <[email protected]> * Remove old code Signed-off-by: Daniel Deluiggi <[email protected]> * Fix test Signed-off-by: Daniel Deluiggi <[email protected]> * update changelog Signed-off-by: Daniel Deluiggi <[email protected]> --------- Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Alex Le <[email protected]> * Create guide doc for partition compaction Signed-off-by: Alex Le <[email protected]> * Update docs/guides/partitioning-compactor.md Co-authored-by: Charlie Le <[email protected]> Signed-off-by: Alex Le <[email protected]> Signed-off-by: Alex Le <[email protected]> * updated doc Signed-off-by: Alex Le <[email protected]> * clean white space Signed-off-by: Alex Le <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alex Le <[email protected]> Signed-off-by: Daniel Sabsay <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: SungJin1212 <[email protected]> Signed-off-by: Charlie Le <[email protected]> Signed-off-by: yeya24 <[email protected]> Signed-off-by: Ahmed Hassan <[email protected]> Signed-off-by: Friedrich Gonzalez <[email protected]> Signed-off-by: Daniel Deluiggi <[email protected]> Signed-off-by: Justin Jung <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Alex Le <[email protected]> Co-authored-by: Alan Protasio <[email protected]> Co-authored-by: Daniel Sabsay <[email protected]> Co-authored-by: Daniel Sabsay <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: SungJin1212 <[email protected]> Co-authored-by: Charlie Le <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Sam McBroom <[email protected]> Co-authored-by: Ahmed Hassan <[email protected]> Co-authored-by: Friedrich Gonzalez <[email protected]> Co-authored-by: Daniel Blando <[email protected]> Co-authored-by: Justin Jung <[email protected]>

pull-request-size bot added the size/XXL label Jan 15, 2025

dosubot bot added the component/compactor label Jan 15, 2025

danielblando approved these changes Jan 16, 2025

View reviewed changes

yeya24 approved these changes Jan 17, 2025

View reviewed changes

yeya24 reviewed Jan 21, 2025

View reviewed changes

pkg/compactor/sharded_block_populator.go Outdated Show resolved Hide resolved

alexqyle added 6 commits January 21, 2025 10:35

Implemented partition compaction end to end with custom compaction li…

47b4161

…fecycle Signed-off-by: Alex Le <[email protected]>

removed unused variable

28ba8fc

Signed-off-by: Alex Le <[email protected]>

tweak test

051e854

Signed-off-by: Alex Le <[email protected]>

tweak test

a490331

Signed-off-by: Alex Le <[email protected]>

refactor according to comments

4eed127

Signed-off-by: Alex Le <[email protected]>

tweak test

be06845

Signed-off-by: Alex Le <[email protected]>

alexqyle force-pushed the partition-compaction-e2e branch from 5e89a40 to be06845 Compare January 21, 2025 18:39

alexqyle added 2 commits January 21, 2025 11:29

check context error inside sharded posting

24ea371

Signed-off-by: Alex Le <[email protected]>

fix lint

8b2b43b

Signed-off-by: Alex Le <[email protected]>

yeya24 approved these changes Jan 21, 2025

View reviewed changes

alexqyle added 2 commits January 21, 2025 14:41

fix integration test for memberlist

0112b39

Signed-off-by: Alex Le <[email protected]>

make compactor initial wait cancellable

c4d04ab

Signed-off-by: Alex Le <[email protected]>

danielblando merged commit c1a2134 into cortexproject:master Jan 21, 2025
17 checks passed

alexqyle deleted the partition-compaction-e2e branch January 23, 2025 21:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hook up partition compaction end to end implementation #6510

Hook up partition compaction end to end implementation #6510

alexqyle commented Jan 15, 2025 •

edited

Loading

danielblando left a comment

yeya24 left a comment

yeya24 Jan 17, 2025

alexqyle Jan 20, 2025

yeya24 Jan 17, 2025

alexqyle Jan 20, 2025

yeya24 Jan 17, 2025

alexqyle Jan 21, 2025

Hook up partition compaction end to end implementation #6510

Hook up partition compaction end to end implementation #6510

Conversation

alexqyle commented Jan 15, 2025 • edited Loading

danielblando left a comment

Choose a reason for hiding this comment

yeya24 left a comment

Choose a reason for hiding this comment

yeya24 Jan 17, 2025

Choose a reason for hiding this comment

alexqyle Jan 20, 2025

Choose a reason for hiding this comment

yeya24 Jan 17, 2025

Choose a reason for hiding this comment

alexqyle Jan 20, 2025

Choose a reason for hiding this comment

yeya24 Jan 17, 2025

Choose a reason for hiding this comment

alexqyle Jan 21, 2025

Choose a reason for hiding this comment

alexqyle commented Jan 15, 2025 •

edited

Loading