This implements a value-based lock in the crypto stores. The intent is to use that for multiple processes to be able to make writes into the store concurrently, while still cooperating on who does them. In particular, we need this for #1928, since we may have up to two different processes trying to write into the crypto store at the same time.

New methods in the CryptoStore trait

The idea is to introduce two new methods touching custom values in the crypto store:

• one to atomically insert a value, only if it was missing (so, not following the semantics of upsert used in the set_custom_value)
• one to atomically remove a custom value

Those two operations match the semantics we want:

• take the lock only if it ain't taken already == insert an entry only if it was missing
• release the lock = remove the entry

By looking at the number of lines affected by the query, we can infer whether the insert/remove happened or not, that is, if we managed to take the lock or not.

High-level APIs

I've also added an high-level API, CryptoStoreLock, that helps managing such a lock, and adds some niceties on top of that:

• exponential backoff to retry attempts at acquiring the lock, when it was already taken
• attempt to gracefully recover when the lock has been taken by an app that's been killed by the environment
• full configuration of the key / value / backoff parameters

I've also attempted to make a CryptoStoreLockGuard, that would be RAII struct taking the lock upon creation and releasing it upon drop, but:

• async Drops are not a thing yet, in native Rust
• even if we spawn a blocking task that runs the cleanup, we cannot ensure that it will be finished before the next attempt to take a lock. Worst case: it's the same holder trying to take the lock (someone else couldn't take it, they would wait), but then we'd consider the lock as acquired (per the graceful recover semantics), and then the deletion would happen, and then we would panic when deleting it.

As a matter of fact, I think we should probably not have the CryptoStoreLockGuard at all, since it's causing more problems than it fixes. I don't know how big an ergonomic issue it will be, though.

Test program

There's also a test program in which I shamelessly show my rudimentary unix skills; I've put it in the labs/ directory but this could as well be a large integration test. A parent program initially fills a custom crypto store, then creates a pipe() for 1-way communication with a child created with fork(); then the parent sends commands to the child. These commands consist in reading and writing into the crypto store, using a lock. And while the child attempts to perform these operations, the parent tries hard to get the lock at the same time. This helps figuring out a few issues and making sure that cross-process locking would work as intended.