Store unregister shutdown

[andrejbranch]

What this PR does: Allows prevention of unregistering store gateways from the ring on shutdown. Allows prevention of replicating store gateway shard blocks when leaving the ring.

Which issue(s) this PR fixes:
Fixes #2823

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Will update change log after feedback

[pstibrany]

Allows prevention of unregistering store gateways from the ring on shutdown. Allows prevention of replicating store gateway shard blocks when leaving the ring.

Keeping store-gateways in the ring isn't enough. When store-gateway restarts, and starts syncing blocks, it sets itself to JOINING state in the ring. Other store-gateways will notice this change and they will start loading the blocks too. See issue #3570 for discussion.

[pstibrany]

Keeping store-gateways in the ring isn't enough. When store-gateway restarts, and starts syncing blocks, it sets itself to JOINING state in the ring. Other store-gateways will notice this change and they will start loading the blocks too. See issue #3570 for discussion.

Scratch that... it may be enough if only single store-gateway is restarting. I need to rethink this.

[bboreham]

@andrejbranch do you think you will come back to work on this?

[andrejbranch]

@bboreham happy to but need some direction

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.