feat: set status code to 500 if unexpected error occurs before streaming in app router (#56236)

<!-- Thanks for opening a PR! Your contribution is much appreciated.
To make sure your PR is handled as smoothly as possible we request that
you follow the checklist sections below.
Choose the right checklist for the change(s) that you're making:

## For Contributors

### Improving Documentation

- Run `pnpm prettier-fix` to fix formatting issues before opening the
PR.
- Read the Docs Contribution Guide to ensure your contribution follows
the docs guidelines:
https://nextjs.org/docs/community/contribution-guide

### Adding or Updating Examples

- The "examples guidelines" are followed from our contributing doc
https://github.com/vercel/next.js/blob/canary/contributing/examples/adding-examples.md
- Make sure the linting passes by running `pnpm build && pnpm lint`. See
https://github.com/vercel/next.js/blob/canary/contributing/repository/linting.md

### Fixing a bug

- Related issues linked using `fixes #number`
- Tests added. See:
https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md

### Adding a feature

- Implements an existing feature request or RFC. Make sure the feature
request has been accepted for implementation before opening a PR. (A
discussion must be opened, see
https://github.com/vercel/next.js/discussions/new?category=ideas)
- Related issues/discussions are linked using `fixes #number`
- e2e tests added
(https://github.com/vercel/next.js/blob/canary/contributing/core/testing.md#writing-tests-for-nextjs)
- Documentation added
- Telemetry added. In case of a feature if it's used or not.
- Errors have a helpful link attached, see
https://github.com/vercel/next.js/blob/canary/contributing.md


## For Maintainers

- Minimal description (aim for explaining to someone not on the team to
understand the PR)
- When linking to a Slack thread, you might want to share details of the
conclusion
- Link both the Linear (Fixes NEXT-xxx) and the GitHub issues
- Add review comments if necessary to explain to the reviewer the logic
behind a change

### What?

### Why?

### How?

Closes NEXT-
Fixes #

-->

This PR therefore introduces to always set response status code to 500
unless it is a `NotFoundError` or `RedirectError`. This PR would fix
issue #56235. See also:
https://codesandbox.io/p/sandbox/nice-panini-2z3mcp .

**Current Behavior**
At the moment, when an unexpected error occurs during app server
rendering, a 200 ok is returned as status code. This seems to be
undesirable because of the success status CDNs will cache the error
pages and crawlers will index the page considering the error content as
the actual content.

**Desired Behavior**
This issue is related to discussion
#53225. Even though I
understand that the response status code cannot be set if streaming has
started, in my view it would be best to set the response status to 500
whenever it can (so before the streaming has started) for SEO and (CDN)
http caching. This would also be consistent with how 404s currently
work; that is, response status code is set to 404 if `NotFoundError`
occurred before streaming (related
[issue](#43831) &
[PR](#55542)).

Ideally, when a runtime error happens after streaming, a `<meta
name="robots" content="noindex" />` would also be added. But I didn't
want to make the PR too complex before receiving feedback.

---------

Co-authored-by: Vũ Văn Dũng <[email protected]>
Co-authored-by: Tobias Koppers <[email protected]>

Author: 3 people

packages/next/src/server/app-render/app-render.tsx

@@ -728,6 +728,9 @@ async function renderToHTMLOrFlightImpl(
         }
 
         const is404 = res.statusCode === 404
+        if (!is404 && !hasRedirectError) {
+          res.statusCode = 500
+        }
 
         // Preserve the existing RSC inline chunks from the page rendering.
         // To avoid the same stream being operated twice, clone the origin stream for error rendering.

test/e2e/app-dir/app/app/(newroot)/dashboard/project/[projectId]/page.js

@@ -1,6 +1,6 @@
 import { use } from 'react'
 
-function getData({ params }) {
+async function getData({ params }) {
   return {
     now: Date.now(),
     params,

test/production/app-dir/unexpected-error/app/isr-unexpected-error/page.tsx

@@ -0,0 +1,9 @@
+export const revalidate = 1
+
+export default function UnexpectedErrorPage(props) {
+  // use query param to only throw error during runtime, not build time
+  if (props.searchParams.error) {
+    throw new Error('Oh no')
+  }
+  return <p id="page">/unexpected-error</p>
+}

test/production/app-dir/unexpected-error/app/layout.tsx

@@ -0,0 +1,7 @@
+export default function Root({ children }: { children: React.ReactNode }) {
+  return (
+    <html>
+      <body>{children}</body>
+    </html>
+  )
+}

test/production/app-dir/unexpected-error/app/ssr-unexpected-error-after-streaming/loading.tsx

@@ -0,0 +1,3 @@
+export default function Loading() {
+  return <p>loading</p>
+}

test/production/app-dir/unexpected-error/app/ssr-unexpected-error-after-streaming/page.tsx

@@ -0,0 +1,7 @@
+export default function UnexpectedErrorPage(props) {
+  // use query param to only throw error during runtime, not build time
+  if (props.searchParams.error) {
+    throw new Error('Oh no')
+  }
+  return <p id="page">/unexpected-error</p>
+}

test/production/app-dir/unexpected-error/app/ssr-unexpected-error/page.tsx

@@ -0,0 +1,7 @@
+export default function UnexpectedErrorPage(props) {
+  // use query param to only throw error during runtime, not build time
+  if (props.searchParams.error) {
+    throw new Error('Oh no')
+  }
+  return <p id="page">/unexpected-error</p>
+}

test/production/app-dir/unexpected-error/next.config.js

@@ -0,0 +1,6 @@
+/**
+ * @type {import('next').NextConfig}
+ */
+const nextConfig = {}
+
+module.exports = nextConfig

test/production/app-dir/unexpected-error/unexpected-error.test.ts

@@ -0,0 +1,26 @@
+import { createNextDescribe } from 'e2e-utils'
+
+createNextDescribe(
+  'unexpected-error',
+  {
+    files: __dirname,
+  },
+  ({ next }) => {
+    it('should set response status to 500 for unexpected errors in ssr app route', async () => {
+      const res = await next.fetch('/ssr-unexpected-error?error=true')
+      expect(res.status).toBe(500)
+    })
+
+    it('cannot change response status when streaming has started', async () => {
+      const res = await next.fetch(
+        '/ssr-unexpected-error-after-streaming?error=true'
+      )
+      expect(res.status).toBe(200)
+    })
+
+    it('should set response status to 500 for unexpected errors in isr app route', async () => {
+      const res = await next.fetch('/isr-unexpected-error?error=true')
+      expect(res.status).toBe(500)
+    })
+  }
+)