account sequence mismatch not handled in `TxClient.SubmitPayForBlob`

[eshaan7]

Summary of Bug

I've implemented a minimal transaction client that imports celestia-app as a package and uses the TxClient.SubmitPayForBlob method to locally sign and submit blob transactions.

Each instance of the client runs with a unique keyring, but all communicate with a single centralized celestia-app bridge node.

The implementation generally works well, but occasionally I encounter the following error:

broadcast tx error: account sequence mismatch: expected 8901, got 8900: incorrect account sequence

When this happens, the only reliable fix seems to be restarting the client (looks to be some kind of a cache issue?)

Version

v3.8.1

Steps to Reproduce

N/A

[evan-forbes]

thanks for submitting this @eshaan7

to confirm

Each instance of the client runs with a unique keyring

does this mean that a unique account was submitting a single tx at a time, waiting for each to be confirmed before submitting another?

unfortunately, this is a widespread issue amongst cosmos-sdk based chains atm. Since the RPC queries from the committed state and not the mempool state, the mempool will reject txs signed with the sequence returned by RPC when the txclient first signs the tx.

[evan-forbes]

retrying to submit the tx after it fails should hack around the problem, but the TxClient leaves the error handling up to the user.

[evan-forbes]

we're scheduled to work on permanent fixes as to never hit this, as UX is a high priority after our current work

[cmwaters]

It's interesting that the expected is ahead of what the client submitted. The client locally increments the sequence. The fact that the expected is ahead means another transaction was submitted that the client was unaware of (unless there's some bug where it might increment twice). This would imply that a transaction was submitted using the same keys but outside of the TxClient or what is possible is that the TxClient is falsely decrementing the sequence because a transaction was locally evicted but not globally and ended up being committed in a block

looks to be some kind of a cache issue

Yeah, when you restart the client you re request the account's sequence from the latest state.

[evan-forbes]

to move a convo from a sync here:

There are a few different potential root problems that result in this error. We have some solutions to known edge cases, but we want to be as sure as we can that we're addressing the specific edge case. That being said, since there is more than one way to hit this bug, there its difficult to know with certainty that we fixed the root cause of this specific issue.

we're either going to add temporary introspection that helps debug this issue, or more permanent introspection that could help debug other known edge cases #4633 .

I'll defer to @yarikbratashchuk on which strategy to take in terms of introspection and replicating this issue. It might be fastest to first focus on debugging this issue, then adding more permanent generally useful introspection later. This will likely mean keeping track of the state of the txclient (such as when the sequence number changes or the results from pings to the rpc while it waits for a tx to be confirmed) and the state of the tx in the mempool (when we actually prune each tx).

To replicate, I'd first try with a simple unit/integration test using testnode and 6s block times and the default ttls. Then I'd see if we could hit the above error with a sequence higher than what was expected by submitting txs constantly. if that doesn't work, then we can try and replicate the issue in a larger testnet, such as in docker (using the new e2e tools or localdevnet on the v3.x branch) or even on mocha

[cmwaters]

My latest hypothesis of the problem is:

The user submits a transaction. It's broadcasted, but then the consensus node evicts it (they have custom evicting logic). The node tells the TxClient that it is evicted and the TxClient rolls back their sequence by one. However afterwards, another node proposed it in a block and then the consensus node updates it's nonce to be ahead of the TxClient

If you wanted to confirm this you'd need to be tracking evictions and you would need to confirm that the tx hash of the tx with the sequence mismatch was evicted by the consensus node and also included in a later block. This isn't something adding metrics will be able to work out by itself

[yarikbratashchuk]

does this mean that a unique account was submitting a single tx at a time, waiting for each to be confirmed before submitting another?

@eshaan7 can you give some light on this?