Compactor interrupted during block upload leaves partial blocks in the storage forever

[pstibrany]

After compactor creates new blocks locally, it starts uploading new blocks to object storage. If compactor is interrupted in the middle of upload (eg. it is asked to shut down), it will stop the upload and leave partial block in the object storage. On restart (before next compaction) compactor will remove blocks from last compaction on disk, but partial blocks in the object storage will stay in the storage forever. Compactor will discover such blocks and will complain about them, but currently it has no logic to delete partial blocks, and we expect that operator investigates what happened.

Compactor could be little smarter, and when it is interrupted during block upload, it could actually try to mark partially uploaded block for deletion. This is safe, because compactor will never attempt to re-upload such block in the future (instead it will run new compaction, producing new block). Marking for deletion would need to be done with new context (possibly with some timeout to avoid blocking shutdown for too long).

Alternative solution may be to have better handling of partial blocks in the storage: When compactor detects partial block, it may "remember" this (in memory), and if this same block is still partial (has no meta.json) after preconfigured period (eg. 6h), compactor could delete the block. With many compactors running, multiple compactors may try to delete the same block after this period, but that's fine.

[osg-grafana]

Is this a bug, something that simply needs to be documented, or both?

[pstibrany]

This is a bug to be fixed. We see it sometimes happening in our production environments. Not frequently, but it happens.

[pracucci]

Have you considered the option to wait until on-going uploads are completed before shutting down the compactor?

[pstibrany]

Have you considered the option to wait until on-going uploads are completed before shutting down the compactor?

I don't think that's an option. Compactor should honor shutdown request as soon as possible, and not delay it by finishing upload which can take a long time. Interrupting upload is unfortunate, but correct reaction.

[pstibrany]

Have you considered the option to wait until on-going uploads are completed before shutting down the compactor?

I don't think that's an option. Compactor should honor shutdown request as soon as possible, and not delay it by finishing upload which can take a long time. Interrupting upload is unfortunate, but correct reaction.

I would rather see finishing the upload upon restart.

[pstibrany]

Have you considered the option to wait until on-going uploads are completed before shutting down the compactor?

I don't think that's an option. Compactor should honor shutdown request as soon as possible, and not delay it by finishing upload which can take a long time. Interrupting upload is unfortunate, but correct reaction.

Discussed with @pracucci offline: "As soon as possible" may be a stretch. Compactor should honor shutdown request in reasonable time. We could introduce "upload finish period" to give compactor extra chance to finish upload. If this times out, compactor would instead mark block for deletion. "Upload finish period" should be shorter than "termination graceful period" used in Kubernetes or similar environments.

[vladimirGuryanov]

"upload finish period" will be helpful for cases with correct shutdown, but it doesn't cover the case with abnormal shutdown. For example, if compactor die by OOM or node with compactor will go out during upload.

Additionally, I reviewed the code and found this constant that sets the period for delete portion blocks, but it's used only in flag help message.

[aknuds1]

I'd like to work on this.

[pstibrany]

#2285 fixes this, when it's enabled.

[aknuds1]

@pstibrany yeah, I just realized the same earlier :)

[pstibrany]

Closing this as #2285 fixes the problem.