Skip to content

[Tracking] Hive ethereum/engine simulation #1285

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
mpaulucci opened this issue Nov 26, 2024 · 0 comments
Open

[Tracking] Hive ethereum/engine simulation #1285

mpaulucci opened this issue Nov 26, 2024 · 0 comments
Assignees
@fedacking
Labels
ef-tests Hive tests, execution-spec-tests L1 Ethereum client

Comments

@mpaulucci
Copy link
Collaborator

mpaulucci commented Nov 26, 2024 •
edited
Loading

These are the failing tests

make run-hive SIMULATION=ethereum/engine TEST_PATTERN="engine-cancun"
cat $(ls -t hive/workspace/logs/*-*.json | head -n1) | jq '.testCases[] | select(.summaryResult.pass == false) | .name'
  • Invalid NewPayload, Incomplete Transactions, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Signature, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Signature, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction Nonce, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Nonce, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction GasPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction GasPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction GasTipCapPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction Gas, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Gas, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction Value, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Value, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction ChainID, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction ChainID, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=False, Invalid P1
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P1
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=False, Invalid P9
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P9
  • Sidechain Reorg
  • Re-Org Back to Canonical Chain From Syncing Chain
  • Re-Org Back into Canonical Chain, Depth=10, Execute Side Payload on Re-Org
  • Invalid PayloadAttributes, Missing BeaconRoot, Syncing=True
  • Invalid NewPayload, ParentBeaconBlockRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, BlobGasUsed, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Blob Count on BlobGasUsed, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, ExcessBlobGas, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, VersionedHashes, Syncing=False, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, VersionedHashes, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, VersionedHashes Version, Syncing=False, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, VersionedHashes Version, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Incomplete VersionedHashes, Syncing=False, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Incomplete VersionedHashes, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Extra VersionedHashes, Syncing=False, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Extra VersionedHashes, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Request Blob Pooled Transactions Single
  • Request Blob Pooled Transactions Multiple
  • Bad Hash on NewPayload (Syncing=true, Sidechain=false)
  • Bad Hash on NewPayload (Syncing=true, Sidechain=true)
  • Unknown HeadBlockHash
  • Invalid PayloadAttributes, Zero timestamp, Syncing=True
  • Blob Transaction Ordering, Multiple Clients
  • Invalid PayloadAttributes, Parent timestamp, Syncing=True
  • In-Order Consecutive Payload Execution
  • Valid NewPayload->ForkchoiceUpdated on Syncing Client
  • Invalid NewPayload, ParentHash, Syncing=False, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, ParentHash, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, StateRoot, Syncing=True, EmptyTxs=True, DynFeeTxs=False
  • Invalid NewPayload, StateRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, ReceiptsRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Number, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, GasLimit, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, GasUsed, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Timestamp, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, PrevRandao, Syncing=True, EmptyTxs=False, DynFeeTxs=False
make run-hive SIMULATION=ethereum/engine TEST_PATTERN="engine-withdrawals"
cat $(ls -t hive/workspace/logs/*-*.json | head -n1) | jq '.testCases[] | select(.summaryResult.pass == false) | .name'

Currently failing tests:

  • Empty Withdrawals #1605
  • Corrupted Block Hash Payload (INVALID) #1608
  • Sync after 2 blocks - Withdrawals on Block 1 - Single Withdrawal Account - No Transactions
  • Sync after 2 blocks - Withdrawals on Block 1 - Single Withdrawal Account
  • Sync after 2 blocks - Withdrawals on Block 2 - Multiple Withdrawal Accounts - No Transactions
  • Sync after 2 blocks - Withdrawals on Block 2 - Multiple Withdrawal Accounts
  • Sync after 128 blocks - Withdrawals on Block 2 - Multiple Withdrawal Accounts
  • Withdrawals Fork on Block 1 - 1 Block Re-Org
  • Withdrawals Fork On Genesis
  • Withdrawals Fork on Block 1 - 8 Block Re-Org NewPayload
  • Withdrawals Fork on Block 1 - 8 Block Re-Org, Sync
  • Withdrawals Fork on Block 8 - 10 Block Re-Org NewPayload
  • Withdrawals Fork on Block 8 - 10 Block Re-Org Sync
  • Withdrawals Fork on Canonical Block 8 / Side Block 7 - 10 Block Re-Org
  • Withdrawals Fork on Canonical Block 8 / Side Block 7 - 10 Block Re-Org Sync
  • Withdrawals Fork on Canonical Block 8 / Side Block 9 - 10 Block Re-Org
  • Withdrawals Fork on Canonical Block 8 / Side Block 9 - 10 Block Re-Org Sync
  • GetPayloadBodiesByRange
  • Withdrawals Fork on Block 1
  • GetPayloadBodies After Sync
  • GetPayloadBodiesByRange (Sidechain)
  • GetPayloadBodiesByRange (Empty Transactions/Withdrawals)
  • GetPayloadBodiesByHash
  • GetPayloadBodiesByHash (Empty Transactions/Withdrawals)
  • GetPayloadBodies Parallel
  • Withdrawals Fork on Block 2
  • Withdrawals Fork on Block 3
  • Withdraw to a single account
  • Withdraw to two accounts
  • Withdraw many accounts
  • Withdraw zero amount
make run-hive SIMULATION=ethereum/engine TEST_PATTERN="engine-api"
cat $(ls -t hive/workspace/logs/*-*.json | head -n1) | jq '.testCases[] | select(.summaryResult.pass == false) | .name'

Current failing tests:

  • Invalid Missing Ancestor Syncing ReOrg, Transaction Signature, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
  • Invalid Missing Ancestor Syncing ReOrg, Transaction Nonce, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
  • Invalid Missing Ancestor Syncing ReOrg, Transaction Gas, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
  • Invalid Missing Ancestor Syncing ReOrg, Transaction GasPrice, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
  • Invalid Missing Ancestor Syncing ReOrg, Transaction Value, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
  • Sidechain Reorg
  • Re-Org Back to Canonical Chain From Syncing Chain
  • Re-Org Back into Canonical Chain, Depth=10, Execute Side Payload on Re-Org
  • Unknown HeadBlockHash
  • Invalid PayloadAttributes, Zero timestamp, Syncing=True
  • Invalid PayloadAttributes, Parent timestamp, Syncing=True
  • In-Order Consecutive Payload Execution
  • Valid NewPayload->ForkchoiceUpdated on Syncing Client
  • Invalid NewPayload, ParentHash, Syncing=False, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, ParentHash, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, StateRoot, Syncing=True, EmptyTxs=True, DynFeeTxs=False
  • Invalid NewPayload, StateRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, ReceiptsRoot, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Bad Hash on NewPayload (Syncing=true, Sidechain=false)
  • Invalid NewPayload, Number, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, GasLimit, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, GasUsed, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Timestamp, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, PrevRandao, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Bad Hash on NewPayload (Syncing=true, Sidechain=true)
  • Invalid NewPayload, Incomplete Transactions, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Signature, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Signature, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction Nonce, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Nonce, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction GasPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction GasPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction GasTipCapPrice, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction Gas, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Gas, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction Value, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction Value, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid NewPayload, Transaction ChainID, Syncing=True, EmptyTxs=False, DynFeeTxs=False
  • Invalid NewPayload, Transaction ChainID, Syncing=True, EmptyTxs=False, DynFeeTxs=True
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=False, Invalid P1
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P1
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=False, Invalid P9
  • Invalid Missing Ancestor ReOrg, StateRoot, EmptyTxs=True, Invalid P9
  • Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=True, CanonicalReOrg=True, Invalid P9
  • Invalid Missing Ancestor Syncing ReOrg, StateRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
  • Invalid Missing Ancestor Syncing ReOrg, ReceiptsRoot, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
  • Invalid Missing Ancestor Syncing ReOrg, GasLimit, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
  • Invalid Missing Ancestor Syncing ReOrg, GasUsed, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
  • Invalid Missing Ancestor Syncing ReOrg, Timestamp, EmptyTxs=False, CanonicalReOrg=True, Invalid P8
  • Invalid Missing Ancestor Syncing ReOrg, Incomplete Transactions, EmptyTxs=False, CanonicalReOrg=True, Invalid P9
@mpaulucci mpaulucci pinned this issue Nov 26, 2024
@lambdaclass lambdaclass deleted a comment from ElFantasma Dec 30, 2024
@mpaulucci mpaulucci added ef-tests Hive tests, execution-spec-tests L1 Ethereum client labels Jan 9, 2025
This was referenced Feb 17, 2025
Merged
Merged
github-merge-queue bot pushed a commit that referenced this issue Feb 19, 2025
@MarcosNicolau
fix(l1): hive engine-cancun suite ForkId tests (#1976)
6bd2f76 
**Motivation**
Fix failing tests in the engine-cancun hive suite.


**Description**
This update fixes all tests matching the pattern `ForkId: *` in the
`engine-cancun` suite. The issue resulted from not filtering out
duplicate fork values and failing to exclude forks that occurred before
the genesis block.

Advances on #1285
github-merge-queue bot pushed a commit that referenced this issue Feb 19, 2025
@MarcosNicolau
fix(l1): hive invalid ancestor (#1926)
ac57e8d 
**Motivation**
Fixes hive invalid ancestor tests under `engine-cancun` with the pattern
`engine-cancun/Invalid Missing .* .*CanonicalReOrg=False`.

**Description**
The following was implemented to fix the tests:
- Because `forkchoice_update` and `new_payload` require us to send the
latest valid hash, we now cache the last valid hash of the chain
corresponding to bad blocks, these are the `invalid_ancestors`.
- The bad blocks are cached when:
   1. When failing to add a block during syncing
   2.  When the block execution of `new_payload` fails.
- The latest valid hash was fixed on `new_payload`; this was a known
issue see #982.
- Finally, when syncing, if we have a pending block from a `new_payload`
request, we attach it if the parent hash of the block corresponds to the
last header parent hash.

Closes #982
Advances on #1285
JereSalo pushed a commit that referenced this issue Feb 28, 2025
@MarcosNicolau @JereSalo
fix(l1): hive engine-cancun suite ForkId tests (#1976)
974c6d5 
**Motivation**
Fix failing tests in the engine-cancun hive suite.

**Description**
This update fixes all tests matching the pattern `ForkId: *` in the
`engine-cancun` suite. The issue resulted from not filtering out
duplicate fork values and failing to exclude forks that occurred before
the genesis block.

Advances on #1285
JereSalo pushed a commit that referenced this issue Feb 28, 2025
@MarcosNicolau @JereSalo
fix(l1): hive invalid ancestor (#1926)
589281f 
**Motivation**
Fixes hive invalid ancestor tests under `engine-cancun` with the pattern
`engine-cancun/Invalid Missing .* .*CanonicalReOrg=False`.

**Description**
The following was implemented to fix the tests:
- Because `forkchoice_update` and `new_payload` require us to send the
latest valid hash, we now cache the last valid hash of the chain
corresponding to bad blocks, these are the `invalid_ancestors`.
- The bad blocks are cached when:
   1. When failing to add a block during syncing
   2.  When the block execution of `new_payload` fails.
- The latest valid hash was fixed on `new_payload`; this was a known
issue see #982.
- Finally, when syncing, if we have a pending block from a `new_payload`
request, we attach it if the parent hash of the block corresponds to the
last header parent hash.

Closes #982
Advances on #1285
@mpaulucci mpaulucci changed the title [Tracking] Hive engine-cancun test suite [Tracking] Hive engine-cancun test suite (Cancun) Apr 7, 2025
@mpaulucci mpaulucci mentioned this issue Apr 10, 2025
Open
@mpaulucci mpaulucci changed the title [Tracking] Hive engine-cancun test suite (Cancun) [Tracking] Hive ethereum/engine simulation Apr 24, 2025
This was referenced Apr 24, 2025
Closed
Closed
@fedacking fedacking self-assigned this Apr 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fedacking fedacking

Labels
ef-tests Hive tests, execution-spec-tests L1 Ethereum client
Projects
ethrex_l1
Status: No status
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mpaulucci @fedacking