Indices deletion in leader cluster do not get replicated in follower cluster

[akolarkunnu]

Description

Introduced a new setting parameter 'plugins.replication.replicate.delete_index'. If this property's value is set to true, it will replicate the deletion of leader index to follower. That means, the corresponding follower index will get deleted automatically. The default behavior is just pause the replication(existing behavior).
The property 'plugins.replication.replicate.delete_index' is follower level. It has to set to each follower to delete index from that follower.
Flow of deletion - Whenever replication state comes to FailedState, first check whether delete_index replication is enabled in the setting and failure reason is "IndexNotFoundException of leader index" then it will trigger the stop of replication. Then in the cleanup() activity of IndexReplicationTask check whether replication is state is COMPLETED and corresponding leader index is deleted, then trigger the deletion of the follower index.
If delete_index replication is not enabled or failure reason is not "IndexNotFoundException of leader index" then it will just pause the replication.

Automated Testing:

• Added test cases to test the deletion/pause of follower index by setting the property 'plugins.replication.replicate.delete_index' to true and false
• There is already a test case to test auto pause scenario in PauseReplicationIT.test auto pause of index replication when leader index is unavailable
  Manual Testing:
• Tested the default scenario - without setting the property
• Tested by setting property to true with single follower (single index and multiple indices)
• Tested by setting property to false with single follower (single index and multiple indices)
• Tested by setting property to true/false with multiple followers with different combinations(single index and multiple indices)

Related Issues

Resolves #1434

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ankitkala]

The flow of deletion is not very clear. Let's also document that in PR description for context.

[akolarkunnu]

The flow of deletion is not very clear. Let's also document that in PR description for context.

Added the deletion flow details in the PR description

[akolarkunnu]

@ankitkala and other maintainers, Gentle Reminder for review!

[ankitkala]

Changes looks good to me. Have few minor comments

[soosinha]

Looks good to me.
One question: After the follower index gets deleted, what does the status API show for the follower index ?

[akolarkunnu]

Looks good to me. One question: After the follower index gets deleted, what does the status API show for the follower index ?

It shows error message as "index_not_found_exception","reason":"no such index [index-id]"
Eg:
curl -XGET "http://${FOLLOWER}/follower-21/_stats"

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [follower-21]","index":"follower-21","resource.id":"follower-21","resource.type":"index_or_alias","index_uuid":"na"}],"type":"index_not_found_exception","reason":"no such index [follower-21]","index":"follower-21","resource.id":"follower-21","resource.type":"index_or_alias","index_uuid":"na"},"status":404}

[soosinha]

What about the metadata in .replication-metadata-store index ? It should show replication as STOPPED for this index.
You can do a normal search on this index to check.

curl -XGET "http://${FOLLOWER}/.replication-metadata-store/_search"

[akolarkunnu]

What about the metadata in .replication-metadata-store index ? It should show replication as STOPPED for this index. You can do a normal search on this index to check.

curl -XGET "http://${FOLLOWER}/.replication-metadata-store/_search"

Output before deletion of leader index:
curl -XGET "http://${FOLLOWER}/.replication-metadata-store/_search"
{"took":4,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":1,"relation":"eq"},"max_score":1.0,"hits":[{"_index":".replication-metadata-store","_id":"follower-21","_score":1.0,"_source":{"connection_name":"leader-cluster","metadata_type":"INDEX","overall_state":"RUNNING","reason":"User initiated","follower_context":{"resource":"follower-21"},"leader_context":{"resource":"leader-02"},"settings":{}}}]}}

Output after deletion of leader index:
curl -XGET "http://${FOLLOWER}/.replication-metadata-store/_search"
{"took":1,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":0,"relation":"eq"},"max_score":null,"hits":[]}}