Skip to content

fix: fixes flaky CI tests in RPC Batch 2 #3495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Mar 17, 2025
Merged

fix: fixes flaky CI tests in RPC Batch 2 #3495

merged 7 commits into from
Mar 17, 2025

Conversation

simzzz
Copy link
Contributor

@simzzz simzzz commented Feb 18, 2025

Description:
The beforeAll() block in the eth_getCode suite is timing out.

By optimizing the hook in the RPC API batch 2 and removing unnecessary and expensive calls, and then rerunning the job multiple times - now it fails much less often.

Related issue(s):

Fixes partially #3353

Notes for reviewer:

Checklist

@simzzz simzzz self-assigned this Feb 18, 2025
@simzzz simzzz added the bug Something isn't working label Feb 18, 2025
@simzzz simzzz added this to the 0.67.0 milestone Feb 18, 2025
Copy link

github-actions bot commented Feb 18, 2025

Test Results

 23 files  +  4  303 suites  +49   48m 4s ⏱️ + 11m 35s
620 tests + 10  612 ✅ +  8  4 💤 ±0  4 ❌ +2 
902 runs  +175  891 ✅ +172  6 💤 ±0  5 ❌ +3 

For more details on these failures, see this check.

Results for commit edba11b. ± Comparison against base commit ef2d8bb.

This pull request removes 2 and adds 12 tests. Note that renamed tests count towards both.
"before all" hook in "@erc20 Acceptance Tests" ‑ RPC Server Acceptance Tests Acceptance tests @erc20 Acceptance Tests "before all" hook in "@erc20 Acceptance Tests"
should execute "web3_client_version" ‑ RPC Server Acceptance Tests Acceptance tests @api-batch-2 RPC Server Acceptance Tests @release Hardcoded RPC Endpoints should execute "web3_client_version"
"after all" hook for "@release should execute "eth_estimateGas" for contract call, using a standard websocket" ‑ RPC Server Acceptance Tests Acceptance tests @web-socket-batch-1 eth_estimateGas "after all" hook for "@release should execute "eth_estimateGas" for contract call, using a standard websocket"
"before all" hook for "Function calling HederaTokenService.isToken(token)" ‑ RPC Server Acceptance Tests Acceptance tests @api-batch-3 RPC Server Acceptance Tests eth_call with contract that calls precompiles "before all" hook for "Function calling HederaTokenService.isToken(token)"
@release Should return errors for blacklisted methods ‑ RPC Server Acceptance Tests Acceptance tests @api-batch-3 RPC Server Acceptance Tests Batch Request Test Suite BATCH_REQUESTS_ENABLED = true @release Should return errors for blacklisted methods
Should allow different accounts associated in the same HbarSpendingPlan to contribute to the same budget ‑ RPC Server Acceptance Tests Acceptance tests @hbarlimiter HBAR Limiter Acceptance Tests HBAR Rate Limit Tests HBAR Rate Limit For Different Spending Plan Tiers @hbarlimiter-batch2 Preconfigured Tiers EXTENDED Tier Should allow different accounts associated in the same HbarSpendingPlan to contribute to the same budget
Should allow different accounts associated in the same HbarSpendingPlan to contribute to the same budget ‑ RPC Server Acceptance Tests Acceptance tests @hbarlimiter HBAR Limiter Acceptance Tests HBAR Rate Limit Tests HBAR Rate Limit For Different Spending Plan Tiers @hbarlimiter-batch2 Preconfigured Tiers PRIVILEGED Tier Should allow different accounts associated in the same HbarSpendingPlan to contribute to the same budget
Should eventually exhaust the hbar limit for EXTENDED user and still allow another EXTENDED user to make calls ‑ RPC Server Acceptance Tests Acceptance tests @hbarlimiter HBAR Limiter Acceptance Tests HBAR Rate Limit Tests HBAR Rate Limit For Different Spending Plan Tiers @hbarlimiter-batch2 Preconfigured Tiers EXTENDED Tier Should eventually exhaust the hbar limit for EXTENDED user and still allow another EXTENDED user to make calls
Should eventually exhaust the hbar limit for PRIVILEGED user and still allow another PRIVILEGED user to make calls ‑ RPC Server Acceptance Tests Acceptance tests @hbarlimiter HBAR Limiter Acceptance Tests HBAR Rate Limit Tests HBAR Rate Limit For Different Spending Plan Tiers @hbarlimiter-batch2 Preconfigured Tiers PRIVILEGED Tier Should eventually exhaust the hbar limit for PRIVILEGED user and still allow another PRIVILEGED user to make calls
Should increase the amount spent of the spending plan by the transaction cost ‑ RPC Server Acceptance Tests Acceptance tests @hbarlimiter HBAR Limiter Acceptance Tests HBAR Rate Limit Tests HBAR Rate Limit For Different Spending Plan Tiers @hbarlimiter-batch2 Preconfigured Tiers EXTENDED Tier Should increase the amount spent of the spending plan by the transaction cost
Should increase the amount spent of the spending plan by the transaction cost ‑ RPC Server Acceptance Tests Acceptance tests @hbarlimiter HBAR Limiter Acceptance Tests HBAR Rate Limit Tests HBAR Rate Limit For Different Spending Plan Tiers @hbarlimiter-batch2 Preconfigured Tiers PRIVILEGED Tier Should increase the amount spent of the spending plan by the transaction cost
should create a BASIC spending plan for a new user and use the same plan on second transaction and different plan on third transaction from another user ‑ RPC Server Acceptance Tests Acceptance tests @hbarlimiter HBAR Limiter Acceptance Tests HBAR Rate Limit Tests HBAR Rate Limit For Different Spending Plan Tiers @hbarlimiter-batch2 BASIC Tier should create a BASIC spending plan for a new user and use the same plan on second transaction and different plan on third transaction from another user
…

♻️ This comment has been updated with latest results.

Signed-off-by: Simeon Nakov <[email protected]>
Signed-off-by: Simeon Nakov <[email protected]>
Signed-off-by: Simeon Nakov <[email protected]>
Signed-off-by: Simeon Nakov <[email protected]>
@simzzz simzzz marked this pull request as ready for review February 21, 2025 13:55
@simzzz simzzz requested review from Nana-EC and a team as code owners February 21, 2025 13:55
quiet-node
quiet-node previously approved these changes Feb 26, 2025
Copy link
Contributor

@quiet-node quiet-node left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice attempt! Would it be possible to also take this opportunity to fix the should execute "eth_sendRawTransaction" if the receiver's account has receiver_sig_required disabled' test? I’ve noticed it fails quite often. If you think it’s better addressed in a separate PR, that works too.

@simzzz
Copy link
Contributor Author

simzzz commented Mar 4, 2025

@quiet-node Yes, I did debug that one as well but I still haven't determined an exact cause. I will get back to it in a separate PR

@simzzz simzzz changed the title fix: fixes flaky CI tests fix: fixes flaky CI tests in RPC Batch 2 Mar 5, 2025
@quiet-node quiet-node modified the milestones: 0.67.0, 0.68.0 Mar 12, 2025
Copy link
Contributor

@konstantinabl konstantinabl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few comments

Copy link
Contributor

@quiet-node quiet-node left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks for looking into this definitely causes time and effort to re-run the CI

@quiet-node
Copy link
Contributor

quiet-node commented Mar 14, 2025

@simzzz woops Batch2 still fails pretty consistently

@simzzz simzzz merged commit e15b5e0 into main Mar 17, 2025
45 of 47 checks passed
@simzzz simzzz deleted the 3353-flaky-ci-fix branch March 17, 2025 13:20
Copy link

codecov bot commented Mar 17, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.64%. Comparing base (4406824) to head (edba11b).
Report is 27 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3495      +/-   ##
==========================================
+ Coverage   85.17%   85.64%   +0.47%     
==========================================
  Files          69       75       +6     
  Lines        4734     4834     +100     
  Branches      999      998       -1     
==========================================
+ Hits         4032     4140     +108     
+ Misses        409      405       -4     
+ Partials      293      289       -4     
Flag Coverage Δ
config-service 93.58% <ø> (-1.58%) ⬇️
relay 79.31% <ø> (-0.24%) ⬇️
server 85.15% <ø> (+1.55%) ⬆️
ws-server 36.31% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 75 files with indirect coverage changes

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants