feat: Prepare Additional Conformity Test Scenarios (#3886) #3887

mwb-al · 2025-07-01T13:04:36Z

Description:

This PR refactors the conformity test suite (@conformity-batch-1) to improve code organization, readability, and maintainability. The main changes include:

What Was Removed

Redundant approaches for overriding Ethereum endpoints

What Was Added

New utils directory with modular utility functions:
- utils.ts - Core utility functions
- constants.ts - Test constants and configuration
- interfaces.ts - Type definitions and interfaces
- overwrites.ts - Override handling logic
- processors.ts - Test data processing functions
- validations.ts - Validation utilities
- transactions.ts - Transaction-related utilities

Why These Changes Were Made

Code Organization: Moved scattered test logic into organized, reusable modules
Improved Readability: Separated concerns into logical modules
Eliminated Redundancy: Unified the approach for handling Ethereum endpoint overrides (overwrites.ts)
Better Maintainability: Modular structure makes it easier to maintain and extend tests

Validation Types in Override Files

The /overwrites folder contains various types of test validations:

eth_blockNumber

eth_blockNumber/simple-test.io - Simple block number test with wildcard result validation

eth_call

eth_call/call-callenv.io - Contract call with environment variables, wildcard result
eth_call/call-contract.io - Basic contract call test
eth_call/call-revert-abi-error.io - Contract call with ABI error revert
eth_call/call-revert-abi-panic.io - Contract call with ABI panic revert
eth_call/call-callenv-options-eip1559.io - Contract call with EIP-1559 options, wildcard result

eth_chainId

eth_chainId/get-chain-id.io - Chain ID retrieval test

eth_createAccessList

eth_createAccessList/create-al-contract.io - Create access list for contract interaction
eth_createAccessList/create-al-value-transfer.io - Create access list for value transfer
eth_createAccessList/create-al-contract-eip1559.io - Create access list for contract with EIP-1559

eth_estimateGas

eth_estimateGas/estimate-gas-contract.io - Gas estimation for contract calls
eth_estimateGas/estimate-gas-value-transfer.io - Gas estimation for value transfers
eth_estimateGas/estimate-gas-contract-eip1559.io - Gas estimation for EIP-1559 contracts

eth_feeHistory

eth_feeHistory/fee-history.io - Fee history with wildcards for gas ratios and base fees

eth_getBalance

eth_getBalance/get-balance-blockhash.io - Balance at specific block hash
eth_getBalance/get-balance.io - Standard balance retrieval
eth_getBalance/get-balance-unknown-account.io - Balance for non-existent account

eth_getBlockByHash

eth_getBlockByHash/get-block-by-hash.io - Block retrieval by hash

eth_getBlockByNumber

eth_getBlockByNumber/get-block-n.io - Block retrieval with multiple wildcard fields
eth_getBlockByNumber/get-finalized.io - Finalized block with hash and timestamp wildcards
eth_getBlockByNumber/get-latest.io - Latest block with dynamic fields
eth_getBlockByNumber/get-genesis.io - Genesis block with dynamic fields
eth_getBlockByNumber/get-safe.io - Safe block with wildcard result

eth_getBlockTransactionCountByNumber

eth_getBlockTransactionCountByNumber/get-genesis.io - Genesis block transaction count with wildcard

eth_getCode

eth_getCode/get-code-contract.io - Contract code retrieval
eth_getCode/get-code-empty.io - Empty contract code test

eth_getLogs

eth_getLogs/no-topics.io - Log query with invalid address parameter error
eth_getLogs/contract-addr.io - Log query with invalid topics parameter error

eth_getProof

eth_getProof/get-account-proof-blockhash.io - Account proof with unsupported method error

eth_getStorageAt

eth_getStorageAt/get-storage.io - Contract storage retrieval
eth_getStorageAt/get-storage-invalid-key.io - Invalid storage key error

eth_getTransactionByBlockHashAndIndex

eth_getTransactionByBlockHashAndIndex/get-transaction-by-block-hash-and-index.io - Transaction by block hash and index

eth_getTransactionByBlockNumberAndIndex

eth_getTransactionByBlockNumberAndIndex/get-transaction-by-block-number-and-index.io - Transaction by block number and index

eth_getTransactionByHash

eth_getTransactionByHash/get-transaction-by-hash.io - Transaction retrieval by hash

eth_getTransactionCount

eth_getTransactionCount/get-nonce.io - Account nonce retrieval

eth_getTransactionReceipt

eth_getTransactionReceipt/get-access-list.io - Transaction receipt for access list transactions
eth_getTransactionReceipt/get-blob-tx.io - Transaction receipt for blob transactions
eth_getTransactionReceipt/get-dynamic-fee.io - Transaction receipt for dynamic fee transactions
eth_getTransactionReceipt/get-legacy-contract.io - Transaction receipt for legacy contract transactions
eth_getTransactionReceipt/get-legacy-input.io - Transaction receipt for legacy input transactions
eth_getTransactionReceipt/get-legacy-receipt.io - Transaction receipt for legacy receipt transactions

eth_sendRawTransaction

eth_sendRawTransaction/send-access-list-transaction.io - Send access list transaction
eth_sendRawTransaction/send-blob-tx.io - Send blob transaction
eth_sendRawTransaction/send-dynamic-fee-access-list-transaction.io - Send dynamic fee access list transaction
eth_sendRawTransaction/send-dynamic-fee-transaction.io - Send dynamic fee transaction
eth_sendRawTransaction/send-legacy-transaction.io - Send legacy transaction

The list includes all endpoints that required modifications - even if only due to the fact that the transactions from chain.rlp are not replayed before the tests start - in order for them to pass successfully.

Related issue(s):
Fixes #3886

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

…se validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…h wildcard validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…ith wildcard validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

Signed-off-by: Michał Walczak <[email protected]>

…hance flexibility (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…tilities (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…maintainability (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…iero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…nd wildcard validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…ion (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…orted method responses (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…sults (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…lob-related fields and improved wildcard handling (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…er handling and added overrides for enhanced modularity (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

… expand transaction receipt test overrides (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

Signed-off-by: Michał Walczak <[email protected]>

… and modular improvements (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

Signed-off-by: Michał Walczak <[email protected]>

…reasons for unsupported EIPs (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…plexTypes` logic for improved object/array handling (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

codecov · 2025-07-09T10:51:33Z

Codecov Report

All modified and coverable lines are covered by tests ✅

@@           Coverage Diff           @@
##             main    #3887   +/-   ##
=======================================
  Coverage        ?   48.12%           
=======================================
  Files           ?       83           
  Lines           ?     4819           
  Branches        ?      989           
=======================================
  Hits            ?     2319           
  Misses          ?     2122           
  Partials        ?      378

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

simzzz · 2025-07-09T11:03:33Z

packages/server/tests/acceptance/data/conformity/utils/validations.ts

+import { ErrorResponse, JsonRpcResponse, Method, Schema } from './interfaces';
+
+// eslint-disable-next-line @typescript-eslint/no-var-requires
+const execApisOpenRpcData = require('../../../../../../../openrpc_exec_apis.json');


Seems like this file doesn't exist yet when the CI is ran and it is causing it to fial, perhaps you can load it lazily here?

Fixed: d1d61e1

simzzz · 2025-07-09T11:07:41Z

packages/server/tests/acceptance/data/conformity/utils/utils.ts

+    if (needError) {
+      return {
+        jsonrpc: '2.0',
+        id: request.id,
+        error: {
+          code: -32603,
+          message: error instanceof Error ? error.message : 'An unknown error occurred',
+        },
+      } as JsonRpcResponse;


Why not return the actual error if such exists? e.g.

if (error.response?.data) { return error.response.data; }

Fixed: 3202fac

simzzz · 2025-07-09T11:21:10Z

packages/server/tests/acceptance/data/conformity/utils/processors.ts

+      expect(valid).to.be.true;
+      if (response.result) {
+        console.log('Comparing response result with expected result.');
+        expect(response.result).to.be.equal(JSON.parse(content.response).result);


nit: isn't this expect redundant because of the isResponseValid call from earlier?

Fixed: b2d3aa7

simzzz · 2025-07-09T11:47:03Z

...ages/server/tests/acceptance/data/conformity/overwrites/eth_getTransactionCount/get-nonce.io

+// All other fields must remain unchanged to preserve the integrity of the original test case.
+
+>> {"jsonrpc":"2.0","id":1,"method":"eth_getTransactionCount","params":["0xc37f417fA09933335240FCA72DD257BFBdE9C275","latest"]}
+<< {"jsonrpc":"2.0","id":1,"result":"0x7"}


Relying on an exact nonce here could make the test fragile in the future, maybe we could use the wildcard here as well and only verify that the response is in a valid nonce format.

Fixed: a31f265

konstantinabl · 2025-07-10T08:29:41Z

Hi there, thanks for the great work!

This PR includes a large number of file changes and added lines, so it would be helpful to expand the description with more context.

Specifically:

Could you provide a summary of what was changed in the conformityTest.spec file (e.g., what was removed, what was added, and why) I see you created new files and exported functions, but you could just point this out in the description?

Also, please clarify why new tests were added—what functionality or edge cases are they covering?

It would be helpful to list the new tests briefly so reviewers can quickly understand their scope and purpose.

konstantinabl · 2025-07-10T11:25:51Z

packages/server/tests/acceptance/data/conformity/utils/validations.ts

+  return checkObjectProperties(actual as Record<string, unknown>, expected as Record<string, unknown>, wildcards, path);
+}
+
+function checkValues(actual: unknown, expected: unknown, wildcards: string[], path = ''): boolean {


Maybe a better name for this is areValuesMatching or something like that, wdyt?

Fixed: 2ef979a

konstantinabl · 2025-07-10T11:26:32Z

packages/server/tests/acceptance/data/conformity/utils/validations.ts

+  return checkObjectProperties(actual as Record<string, unknown>, expected as Record<string, unknown>, wildcards, path);
+}
+
+function checkValues(actual: unknown, expected: unknown, wildcards: string[], path = ''): boolean {


add JSDoc for this method to better clarify what it does

Fixed: 2ef979a

konstantinabl · 2025-07-10T11:28:04Z

packages/server/tests/acceptance/data/conformity/utils/overwrites.ts

+import { legacyTransaction, transaction1559, transaction1559_2930, transaction2930 } from './transactions';
+import { getTransactionCount } from './utils';
+
+export async function updateRequestParams<T = unknown>(


Fixed: db1d6f4

konstantinabl · 2025-07-10T11:28:46Z

packages/server/tests/acceptance/data/conformity/utils/processors.ts

+import { checkResponseFormat, findSchema, isResponseValid } from './validations';
+
+export function splitReqAndRes(content: string) {
+  /**


Why is the JSDoc inside the method, I know usually it is outside. May not be wrong, just wondering

Fixed: 213fa2a

konstantinabl · 2025-07-10T11:31:00Z

packages/server/tests/acceptance/data/conformity/utils/validations.ts

+const ajv = new Ajv({ strict: false });
+addFormats(ajv);
+
+export function checkResponseFormat(


This method is used here https://github.com/hiero-ledger/hiero-json-rpc-relay/pull/3887/files#diff-91e05f05522a20e9c472bbe550e0d72948fe90f252ff853e9aed3ba95a90f833R63 where it is assigned to a variable 'hasMissingKeys' maybe its better to change the name to something like isFormatValid/hasResponseFormatIssues?

Fixed: df81ef7

konstantinabl · 2025-07-10T11:31:50Z

packages/server/tests/acceptance/data/conformity/utils/validations.ts

+
+const ajv = new Ajv({ strict: false });
+addFormats(ajv);
+


Add JSDoc for the checkResponseFormat method

Fixed: df81ef7

konstantinabl · 2025-07-10T11:35:36Z

packages/server/tests/acceptance/data/conformity/utils/validations.ts

+  return false;
+}
+
+function checkComplexTypes(actual: object | null, expected: object, wildcards: string[], path: string): boolean {


Again maybe the name is slightly misleading, since we return true for non-valid results

Fixed: a19ed69

konstantinabl · 2025-07-10T11:41:00Z

packages/server/tests/acceptance/data/conformity/overwrites/eth_call/call-callenv.io

@@ -23,5 +23,8 @@
 // 6. tx.origin        — 0x00 expected and returned; tx sent from 0x0, so origin is also zero.
 // 7. msg.value        — 0x00 expected and returned; no value was sent with the call, as expected.
 //
+
+## wildcard: result


@mwb-al So what exactly is the purpose of the wildcard, to not actually check the values?

As noted in the description, most of these values (especially block.number, block.coinbase, difficulty/prevrandao) vary with each run and chain state. Hence, the wildcard is required to ensure the test remains valid.

…iero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…RPC parameter overrides (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…th detailed parameter and operation descriptions (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…update references with improved JSDoc documentation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…y, improve JSDoc descriptions (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

Signed-off-by: Michał Walczak <[email protected]>

…iero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

…ger#3886) Signed-off-by: Michał Walczak <[email protected]>

Signed-off-by: Michał Walczak <[email protected]>

mwb-al added 19 commits July 1, 2025 14:35

feat: enhance conformity tests to support wildcards in request/respon…

539c303

…se validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test override for eth_getTransactionByBlockHashAndIndex wit…

ee7aaa3

…h wildcard validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test override for eth_getTransactionByBlockNumberAndIndex w…

801bf46

…ith wildcard validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test override for eth_getTransactionCount (hiero-ledger#3886)

1e7d35b

Signed-off-by: Michał Walczak <[email protected]>

feat: add test overrides for eth_sendRawTransaction (hiero-ledger#3886)

accab0e

Signed-off-by: Michał Walczak <[email protected]>

feat: refactor conformity tests to modularize helper utilities and en…

f8e63ce

…hance flexibility (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: refactor conformity tests to extract processors into separate u…

ac9f8ae

…tilities (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: refactor conformity test utilities for improved modularity and …

7738e66

…maintainability (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test override for eth_blockNumber with wildcard validation (h…

82b0cf8

…iero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test overrides for eth_call with updated expected results a…

d86eaa4

…nd wildcard validation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test override for eth_chainId with updated chain ID validat…

e53c568

…ion (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test overrides for eth_createAccessList with updated unsupp…

4bccaeb

…orted method responses (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test overrides for eth_estimateGas with updated expected re…

04e8b5c

…sults (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test override for eth_feeHistory with updated unsupported b…

54ec448

…lob-related fields and improved wildcard handling (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: refactor conformity test utilities with updated request paramet…

6d6072d

…er handling and added overrides for enhanced modularity (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: refactor overwrite handling with async parameter resolution and…

09b13a8

… expand transaction receipt test overrides (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: add test overrides (hiero-ledger#3886)

ab7e196

Signed-off-by: Michał Walczak <[email protected]>

feat: refactor conformity test utilities with strict TypeScript types…

120fcc6

… and modular improvements (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: document reasons for test overrides (hiero-ledger#3886)

88060ff

Signed-off-by: Michał Walczak <[email protected]>

mwb-al marked this pull request as ready for review July 7, 2025 14:28

mwb-al requested review from a team as code owners July 7, 2025 14:28

mwb-al requested a review from simzzz July 7, 2025 14:28

mwb-al and others added 3 commits July 7, 2025 16:28

Merge branch 'main' into 3886_additional-conformity-tests

4926c5e

feat: update test overrides for eth_sendRawTransaction with detailed …

03385e1

…reasons for unsupported EIPs (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

feat: refactor conformity validation utilities with modular `checkCom…

2fa4633

…plexTypes` logic for improved object/array handling (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

simzzz reviewed Jul 9, 2025

View reviewed changes

konstantinabl requested review from a team and konstantinabl July 10, 2025 08:15

konstantinabl requested changes Jul 10, 2025

View reviewed changes

mwb-al added 9 commits July 11, 2025 10:34

chore: rename checkValues to areValuesMatching and improve JSDoc (h…

2ef979a

…iero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

chore: add JSDoc for updateRequestParams function to document JSON-…

db1d6f4

…RPC parameter overrides (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

chore: improve JSDoc for splitReqAndRes and processFileContent wi…

213fa2a

…th detailed parameter and operation descriptions (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

chore: rename checkResponseFormat to hasResponseFormatIssues and …

df81ef7

…update references with improved JSDoc documentation (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

chore: rename validation utility functions for clarity and consistenc…

a19ed69

…y, improve JSDoc descriptions (hiero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

chore: lazy load execApisOpenRpcData (hiero-ledger#3886)

d1d61e1

Signed-off-by: Michał Walczak <[email protected]>

chore: handle Axios error response data in conformity test utility (h…

3202fac

…iero-ledger#3886) Signed-off-by: Michał Walczak <[email protected]>

chore: clean up redundant check in conformity test utility (hiero-led…

b2d3aa7

…ger#3886) Signed-off-by: Michał Walczak <[email protected]>

chore: update eth_getTransactionCount test (hiero-ledger#3886)

a31f265

Signed-off-by: Michał Walczak <[email protected]>

feat: Prepare Additional Conformity Test Scenarios (#3886) #3887

Are you sure you want to change the base?

feat: Prepare Additional Conformity Test Scenarios (#3886) #3887

Uh oh!

Conversation

mwb-al commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Was Removed

What Was Added

Why These Changes Were Made

Validation Types in Override Files

eth_blockNumber

eth_call

eth_chainId

eth_createAccessList

eth_estimateGas

eth_feeHistory

eth_getBalance

eth_getBlockByHash

eth_getBlockByNumber

eth_getBlockTransactionCountByNumber

eth_getCode

eth_getLogs

eth_getProof

eth_getStorageAt

eth_getTransactionByBlockHashAndIndex

eth_getTransactionByBlockNumberAndIndex

eth_getTransactionByHash

eth_getTransactionCount

eth_getTransactionReceipt

eth_sendRawTransaction

Uh oh!

codecov bot commented Jul 9, 2025

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

konstantinabl commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mwb-al commented Jul 1, 2025 •

edited

Loading

konstantinabl commented Jul 10, 2025 •

edited

Loading