Replication stops working for a particular k/v secrets

[udhaya123]

Hi, We have identified a usecase where consul-replicate abruptly stops syncing a particular k/v secret from one data centre to another. We'd like to understand if this is expected behaviour or if it should be considered a bug.

Scenario explantation:
Primary Datacenter: DC1
Disaster Recovery Datacenter: DC2
Consul-replicate configure to replicate data from DC1 -> DC2

The Consul-replicate services runs on one of the two vault application servers in DC2. The kv pairs are successfully replicated from DC1 to DC2. For example, when we create a new secret in DC1, Delete a secret or edit a secret, it successfully replicates to DC2.

However, the data replication for a particular k/v pair abruptly stops working if:

1. We log on to Vault via the UI on DC2 to query a replicated secret. i.e to check if a key and its value has been successfully replicated across to DC2.
2. Then go back DC1 and edit the keyname for this particular k/v pair.

From that point onward the replication for this k/v stops working. Key or password value changes stop replicating across to DC2 for this particular k/v pair.

The current version of consul-replicate is v0.3.1, we tried upgrading it to version v0.4.0 but the issue remains the same.

We have observed the same behaviour consistently across different vault and consul versions,
Production setup:
vault v0.11.6
consul v1.2.4

Test setup:
Vault v1.4.2
Consul v1..6.9

The issues is the same in prod and test.
Any help/guidance would be appreciated.

Note: Once we restart vault in DC2, the replication seems to work ok for all the k/v pairs.

Thanks,