Accounts stuck in weird state after `debug_setHead`

[antazoey]

System information

Geth version: 1.10.26-stable
CL client & version: e.g. lighthouse/nimbus/prysm@v1.0.0 ?
OS & Version: macOS Ventura 13.0
Commit hash : (if develop)

Expected behaviour

Note: I am using a private PoA chain for testing purposes.

I am trying to have isolation in my tests via debug_setHead and using block numbers as the snapshot IDs.

Actual behaviour

No nonce is good enough for accounts after debug_setHead. If I use the value from the eth_transactionCount, it says that the nonce is too low. If I try bumping it manually, the transaction won't get picked up.

Steps to reproduce the behaviour

Here is what my genesis looks like:

        genesis_data: Dict = {
            "overwrite": True,
            "coinbase": "0x0000000000000000000000000000000000000000",
            "difficulty": "0x0",
            "extraData": f"0x{'0' * 64}{sealer}{'0' * 130}",
            "config": {
                "chainId": chain_id,
                "gasLimit": 0,
                "homesteadBlock": 0,
                "difficulty": "0x0",
                "eip150Block": 0,
                "eip155Block": 0,
                "eip158Block": 0,
                "byzantiumBlock": 0,
                "constantinopleBlock": 0,
                "petersburgBlock": 0,
                "istanbulBlock": 0,
                "berlinBlock": 0,
                "londonBlock": 0,
                "clique": {"period": 0, "epoch": 30000},
            },
            "alloc": {a.address: {"balance": str(initial_balance)} for a in accounts},
        }

Geth command:

geth \
    --http \
    --http.addr 127.0.0.1 \
    --http.port 5550 \
    --http.api admin,clique,debug,eth,miner,net,personal,shh,txpool,web3,ws \
    --datadir /private/var/folders/hg/17b1zhsn1n3d8b4rgh6ntt_w0000gn/T/tmpp_5odh88/ethereum/local/dev \
    --maxpeers 0 \
    --networkid 1337 \
    --port 30303 \
    --ipcpath /private/var/folders/hg/17b1zhsn1n3d8b4rgh6ntt_w0000gn/T/tmpp_5odh88/ethereum/local/dev/geth.ipc \
    --verbosity 5 \
    --unlock 0 \
    --password <my-home-dir>/default_blockchain_password \
    --nodiscover \
    --mine --miner.threads 1 \
    --allow-insecure-unlock account list

Backtrace

Method: eth_sendRawTransaction, Response: {'jsonrpc': '2.0', 'id': 176, 'error': {'code': -32000, 'message': 'nonce too low'}}

[holiman]

I am trying to have isolation in my tests via debug_setHead and using block numbers as the snapshot IDs.

Well, I think the problem here is that debug_setHead is meant to be a tool to set back your chain head after something has gone horribly wrong, such as:

• Your node had a consensus flaw, and you got stuck on the wrong side of a fork. After update, you need to rewind+replay.
• Your db has inconsistencies regarding the latest state. You rewind to an earlier point where you had the state, and hope for the best.

It was never intended to be used as a snapshotting-tool to jump back in time.

What it sounds like, is that the txpool state tracker does not function ideally, like you've already seen with the comment here:#22247 .

Anyway, what I kind of mean is that yes, we'll fix this and/or accept a PR to get this fixed, but it's not a high-prio usecase for us, because it's borderline abusing the feature. Depending on what you want to achieve, there are probably other ways of doing it.

[antazoey]

Depending on what you want to achieve, there are probably other ways of doing it.

I am really interested in this! Do you have any advice for how to implement a snapshotting system using only Geth?

[holiman]

Do you have any advice for how to implement a snapshotting system using only Geth?

First of all is to run it gcmode=archive. Then it won't do any in-memory pruning, but save all state. Secondly, what do you mean -- do you want to really go back, or do you just want to query data from an earlier point in time?

(Because if your usecase is to actually go back in time , then is really a blockchain the right abstraction?)

[antazoey]

@holiman

I am trying to do the same as evm_snapshot + evm_revert that you can do in Ganache and Hardhat.
The use-case is about achieving isolation in tests.

[holiman]

I never use ganache/hardhat, so 🤷 , have no idea how that works in practice. IMO you could just run whatever you want to on state X, without actually rolling back to it.

[s1na]

I think they want to do as below:

• Start from block N
• Send a few txes (deploy contracts, interact with them)
• Read state
• Go back to block N (as if nothing happened in between)
• Send other txes
  ...

So I guess debug_setHead is the closest thing that comes there. But IMO geth is not really suitable for testing in this way. Why don't you use foundry/hardhat @unparalleled-js? If you want to test on a real-world state they have forking features.

[antazoey]

Why don't you use foundry/hardhat @unparalleled-js? If you want to test on a real-world state they have forking features.

I use Hardhat and Foundry as well. I am a tooling developer; Geth is one of options for local providers for our users with the caveat that there is no snapshotting. I really like Geth because it has the debug_traceCall method. I know Hardhat and Foundry plan on adding this method (foundry-rs/foundry#2782 and NomicFoundation/hardhat#2971), but it is enticing for the gas reporting feature I am working on. Thus, I have been looking into what else I can do to make Geth (--dev) work similarly to the other local provider options we offer.

[holiman]

We probably need to clear/reset the txpool nonce tracking mecanism when setHead happens.