[Feature Request] Integrate Jepsen Testing for Enhanced Fault Tolerance

[laminelam]

Is your feature request related to a problem? Please describe

OpenSearch lacks formal testing under network partitions and various fault conditions which Jepsen specializes in. This testing is crucial for ensuring the database maintains consistency and handles failures as expected.

Describe the solution you'd like

Integrate Jepsen testing into the OpenSearch testing framework to rigorously assess and improve the system's fault tolerance and consistency. This would include setting up scenarios to test network partitions, data consistency, and recovery mechanisms.

Related component

Other

Describe alternatives you've considered

No response

Additional context

No response

[msfroh]

This testing is crucial for ensuring the database maintains consistency and handles failures as expected.

There is an open question of whether OpenSearch is a database.

I would argue that OpenSearch should embrace eventual consistency, rather than trying to maintain ACID-style semantics, especially since Lucene is eventually consistent (since writes are not visible until you open a new reader).

[Pallavi-AWS]

Jepsen also covers durability and fault tolerance is my understanding - can we evaluate using Jepsen for chaos testing?

[msfroh]

Jepsen also covers durability and fault tolerance is my understanding - can we evaluate using Jepsen for chaos testing?

Definitely! I should clarify that I'm not against using Jepsen to test OpenSearch. It's a pretty awesome framework.

[andrross]

I agree with @msfroh about the stance of not expecting OpenSearch to have ACID-style semantics.

I've added the cluster manager label here because we do expect the cluster management/leader election to be consistent and fault tolerant, and a framework like Jepsen could be valuable. There's probably value beyond cluster management as well, such as asserting the behavior of the primary-replica model of shards and that acknowledged writes are not lost (when replicas are configured), etc.

[rajiv-kv]

[Triage Attendees - 1, 2]

Thanks for filing the issue. Integration tests in ClusterManager induce faults at transport (using interceptors / connections) and verify the behavior to assert the correctness ClusterState.
We can probably explore Jepsen if it can induce faults externally to the system and leverage the existing network tests.