Skip to content

[Need Investigation] Intermittent Request Failures on Previewnet with Unknown Server Error Code 567 #3711

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
quiet-node opened this issue Apr 17, 2025 · 1 comment
Open

[Need Investigation] Intermittent Request Failures on Previewnet with Unknown Server Error Code 567 #3711

quiet-node opened this issue Apr 17, 2025 · 1 comment
Labels
bug Something isn't working Team Promoted Issues to be triaged and discussed by committers, maintainers, to be worked on the following sprint

Comments

@quiet-node
Copy link
Contributor

quiet-node commented Apr 17, 2025 •
edited
Loading

Description

Intermittent request failures have been observed on the Previewnet environment (6%-9% of the requests). These requests fail with a status code of 567 (the fallback code for unknown server errors in MirrorNodeClient), indicating the error likely originates in the Relay service rather than the Mirror Node remote server.

Error Symptoms

The error logs show vague and inconsistent messages:

  • Error invoking RPC: Unexpected 'S'
  • Error invoking RPC: Unexpected '<'
    These errors:
  • Occur in rapid succession during short timeframes
  • Are exclusive to the Previewnet environment
  • Cannot be reproduced in local development environments
  • Do not contain sufficient context in logs for immediate diagnosis

Root Cause

The error is caused by HTML content being returned instead of expected JSON responses. At the time of the issue, MN was returning 502 Server Error with the below content:

<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
</body></html>

The error messages Unexpected 'S' and Unexpected '<' occur when the MirrorNodeClient attempts to parse HTML with JSONBigInt.parse() in the Axios response interceptor:

axiosRequestConfig['transformResponse'] = [
  (data) => {
    // if the data is not valid, just return it to stick to the current behaviour
    if (data) {
      return JSONBigInt.parse(data);
    }
    return data;
  },
];

This is a classic case of receiving HTML error pages (starting with <!DOCTYPE html> or <html>) when JSON is expected. In the Previewnet environment, one of the following is occurring:

  • The Mirror Node service is returning HTML error pages during service degradation
  • A proxy/WAF in the request path is intercepting and returning HTML error pages
  • Network configurations specific to Previewnet are causing redirects to HTML status pages

Suggested Fix

  • Enhance error logging to capture the initial portion of failing responses to confirm HTML content
  • Add logic for the parsing to handle the case when the response is an http page instead of valid JSON object. If response is not a valid JSON object, skip parsing so the error can be thrown properly.
@quiet-node quiet-node added bug Something isn't working Team Promoted Issues to be triaged and discussed by committers, maintainers, to be worked on the following sprint labels Apr 17, 2025
@acuarica
Copy link
Contributor

acuarica commented May 8, 2025

When debugging this issue, it's important to note that the logs are misleading. They look like the following

Error encountered while communicating with the mirror node server: method=GET, path=blocks?limit=1&order=desc, status=567

which implies that the status=567 actually comes from the MN. However, as pointed out in the issue, that's not the case. Status 567 is an internal error code

hiero-json-rpc-relay/packages/relay/src/lib/clients/mirrorNodeClient.ts

Line 395 in 9022680

MirrorNodeClient.unknownServerErrorHttpStatusCode; // Use custom 567 status code as fallback

@github-project-automation github-project-automation bot moved this from Backlog to Done in Smart Contract Sprint Board May 15, 2025
@Ferparishuertas Ferparishuertas moved this from Done to Backlog in Smart Contract Sprint Board May 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Team Promoted Issues to be triaged and discussed by committers, maintainers, to be worked on the following sprint
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Ferparishuertas @acuarica @quiet-node