Run headless tests on `todoApp` production build #2707

infomiho · 2025-04-28T15:29:01Z

Updates the todoApp to be able to run headless tests in the CI.

Closes #884

Copilot

Pull Request Overview

This PR enables headless testing on the todoApp production build by introducing a manual email verification step and updating the CI configuration. Key changes include:

A new HeadlessVerifyEmails page and corresponding headless server operation for verifying user emails.
Modifications to multiple Playwright test files to include an email verification step.
Updates to the Playwright configuration and CI workflow to support different test modes (dev/build).

Reviewed Changes

Copilot reviewed 9 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
src/pages/HeadlessVerifyEmails.tsx	New page component to manually trigger user email verification.
src/auth/headless.ts	Added server-side operation to iterate and verify user emails.
headless-tests/tests/user-api.spec.ts	Updated test to include email verification after signup.
headless-tests/tests/simple.spec.ts	Added email verification to simple spec tests.
headless-tests/tests/helpers.ts	Introduced performEmailVerification helper for tests.
headless-tests/tests/crud.spec.ts	Integrated email verification call in CRUD tests.
headless-tests/tests/auth-hooks.spec.ts	Updated auth hooks tests with email verification steps.
headless-tests/playwright.config.ts	Added HEADLESS_TEST_MODE variable to toggle test modes.
.github/workflows/waspc-ci.yaml	Expanded CI workflow to run headless tests in both dev and build modes.

Files not reviewed (4)

waspc/examples/todoApp/.env.server.example: Language not supported
waspc/examples/todoApp/.env.server.headless: Language not supported
waspc/examples/todoApp/main.wasp: Language not supported
waspc/examples/todoApp/package-lock.json: Language not supported

Comments suppressed due to low confidence (3)

waspc/examples/todoApp/src/pages/HeadlessVerifyEmails.tsx:5

[nitpick] Consider simplifying the onClick handler by passing verifyAllUserEmails directly instead of wrapping it in an arrow function.

<button onClick={() => verifyAllUserEmails()} className="btn btn-primary">

waspc/examples/todoApp/src/auth/headless.ts:42

[nitpick] If this logging is intended for production, consider using a proper logging utility to ensure sensitive data is handled securely.

console.log('Verifying email for user:', emailProviderId.providerUserId)

waspc/examples/todoApp/headless-tests/playwright.config.ts:3

[nitpick] Add a comment explaining the allowed values ('dev' or 'build') for HEADLESS_TEST_MODE to aid future maintainers.

const HEADLESS_TEST_MODE = process.env.HEADLESS_TEST_MODE ?? 'dev'

FranjoMindek

Small nudges.

FranjoMindek · 2025-04-30T11:21:10Z

waspc/examples/todoApp/headless-tests/tests/helpers.ts

Should we run this test only in production code then?

We can't really run any tests without logging in first. And I think we should be running tests on the production builds because we do it manually. So we need to find a way to verify the email.

The problem here is the email verification flow, so either we "brute force" it like we did here or we come up with a way to have a local SMTP server to get the verification email and click on the verify link. Now that I said it - let me see if I can get that up and running.

I've added a local SMTP server to avoid modifying the todoApp to be able to verify the email: cc4de83 (#2707)

I've meant in this way, which I see you've added. My bad if I had bad wording.

if (process.env.HEADLESS_TEST_MODE === 'dev') { // In dev mode, the emails are verified automatically return }

But the SMTP change is nice.

FranjoMindek · 2025-04-30T11:24:27Z

waspc/examples/todoApp/main.wasp

I wanted to say should we compile this part of the application only when running production code headless tests, but little hard to do without main.wasp.ts. Looking forward to when we transition.

waspc/examples/todoApp/src/auth/headless.ts

infomiho · 2025-05-02T12:37:59Z

wasp-app-runner/src/build/index.ts

@@ -29,6 +30,8 @@ export async function startAppInBuildMode({
    pathToApp,
  });

+  await startLocalSmtpServer();


I'm starting an SMTP server when running in the build mode only. Should we do it in the dev mode as well? I'm not sure what would be the use in dev mode.

I don't really see a reason to do it in dev mode.
In a way this further simulates build environment, which we wanted to do. So I think it fits build only.

infomiho · 2025-05-02T13:00:37Z

waspc/examples/todoApp/headless-tests/tests/helpers.ts

+  if (process.env.HEADLESS_TEST_MODE === 'dev') {
+    // In dev mode, the emails are verified automatically
+    return
+  }


This relies on the todoApp having SKIP_EMAIL_VERIFICATION_IN_DEV in its env vars (which we do have in the .env.server.headless file)

Please add this comment to the code too

infomiho · 2025-05-02T13:02:17Z

wasp-app-runner/src/smtp.ts

+import { spawnWithLog } from "./process.js";
+import { log } from "./logging.js";
+
+export async function startLocalSmtpServer(): Promise<void> {


Do we want to have this in the wasp-app-runner?

Alternatively, we could run the Mailcrab SMTP server just in the Wasp CI.

Having it in the wasp-app-runner gives a unified way to test all the example apps that require email verification.

I'd rather it not depends on an external service.
I think Filip can give more input here when he reviews it.
For me this is okay for testing purposes.

I guess if we're starting the database, we can start this too.

A good rule of thumb is: if the dev mode can simulate the feature (e.g., database with sqlite or wasp start db, email verification with CLI output), then so should the build mode.

FranjoMindek

I have nothing of value to add anymore

sodic

Awesome, looks good to me!

Just take care of this please: #2707 (comment)

sodic · 2025-05-06T09:47:53Z

wasp-app-runner/src/smtp.ts

+import { spawnWithLog } from "./process.js";
+import { log } from "./logging.js";
+
+export async function startLocalSmtpServer(): Promise<void> {


I guess if we're starting the database, we can start this too.

A good rule of thumb is: if the dev mode can simulate the feature (e.g., database with sqlite or wasp start db, email verification with CLI output), then so should the build mode.

sodic · 2025-05-06T09:49:48Z

wasp-app-runner/src/smtp.ts

+    if (exitCode !== 0) {
+      log("smtp-server", "error", `SMTP server exited with code ${exitCode}`);


This is a nice example of localized logs and their prefixes (I talked about that in the other PR).

Usually, I'd recommend all process.exit calls to be at the top level (i.e., bubble up the error and exit at the top). But I think we can let it slide in wasp-app-runner.

sodic · 2025-05-06T09:50:55Z

waspc/examples/todoApp/headless-tests/tests/auth-hooks.spec.ts

@@ -21,7 +22,7 @@ test.describe('auth hooks', () => {
    })

    await expect(page.locator('body')).toContainText(
-      'On Before Signup Hook disallows this email.',
+      'On Before Signup Hook disallows this email.'


Aw man, I liked the trailing comma rule :(

Why'd we lose it?

prettier's all vs es5, comment on universal styles PR if you want to influence this

Fine, I'll fight this battle again.

sodic · 2025-05-06T09:51:42Z

waspc/examples/todoApp/headless-tests/tests/helpers.ts

+  if (process.env.HEADLESS_TEST_MODE === 'dev') {
+    // In dev mode, the emails are verified automatically
+    return
+  }


Please add this comment to the code too

infomiho added 7 commits April 28, 2025 15:52

Initial version

3516d71

Update CI

19addaf

Update README with TODO

128e5ea

Update naming

aa31229

Build mode headless test in CI

d581304

Update SMTP_PORT to be a number

5a1a13e

Verify email for production builds of todoApp

50363ed

infomiho mentioned this pull request Apr 29, 2025

Run wasp build output with wasp-app-runner #2706

Merged

infomiho added 3 commits April 29, 2025 12:33

Update README

4e58d46

Cleanup

3ee0f59

Merge branch 'miho-run-wasp-build-output' into miho-wasp-build-todo-ci

e0200c6

infomiho marked this pull request as ready for review April 29, 2025 11:13

infomiho requested a review from Copilot April 29, 2025 11:16

Copilot AI reviewed Apr 29, 2025

View reviewed changes

infomiho requested review from sodic and FranjoMindek April 29, 2025 14:27

FranjoMindek requested changes Apr 30, 2025

View reviewed changes

infomiho added 6 commits May 2, 2025 12:24

Merge branch 'main' into miho-run-wasp-build-output

58bce23

.env.client support. Build & run client in one fn

5cbcfae

Update naming

6886b23

Improve argument parsing

45eb7b0

Use Mailcrab for email verification

cc4de83

Update naming

e433cba

infomiho commented May 2, 2025

View reviewed changes

Extract Mailcrab API url

fd57e77

infomiho commented May 2, 2025

View reviewed changes

FranjoMindek approved these changes May 5, 2025

View reviewed changes

infomiho added 2 commits May 5, 2025 16:16

Uses Branded types for CLI args and app name (#2716)

2520d71

Merge branch 'miho-run-wasp-build-output' into miho-wasp-build-todo-ci

0eebae1

sodic approved these changes May 6, 2025

View reviewed changes

infomiho added 14 commits May 6, 2025 16:28

PR comments

b96e808

Merge branch 'miho-run-wasp-build-output' into miho-wasp-build-todo-ci

3f1cd0f

Add comment to headless test helpers

b9d2c9d

Initial version

be54a08

Update CI

4f970ee

Update README with TODO

b42819a

Update naming

be226c4

Update README

3e72bf0

Cleanup

d8db663

.env.client support. Build & run client in one fn

a5c035e

Update naming

c54ad95

Improve argument parsing

d8dfe4a

Uses Branded types for CLI args and app name (#2716)

2d51158

PR comments

e5d2552

infomiho force-pushed the miho-run-wasp-build-output branch from b96e808 to e5d2552 Compare May 9, 2025 17:01

infomiho added 2 commits May 9, 2025 19:15

Update bin name

90cf7dc

Merge branch 'miho-run-wasp-build-output' into miho-wasp-build-todo-ci

086351b

Base automatically changed from miho-run-wasp-build-output to miho-runner-build-feature May 19, 2025 09:50

Merge branch 'miho-runner-build-feature' into miho-wasp-build-todo-ci

ad34b84

infomiho changed the base branch from miho-runner-build-feature to main May 19, 2025 10:01

infomiho changed the base branch from main to miho-runner-build-feature May 19, 2025 10:01

infomiho added 3 commits May 19, 2025 12:03

Merge branch 'miho-runner-build-feature' into miho-wasp-build-todo-ci

cc22239

Update websocket test to verify email

cfe858f

Start client first

c544a16

infomiho force-pushed the miho-wasp-build-todo-ci branch from 24d319d to c544a16 Compare May 19, 2025 11:39

infomiho merged commit a05cd76 into miho-runner-build-feature May 19, 2025
5 checks passed

infomiho deleted the miho-wasp-build-todo-ci branch May 19, 2025 11:54

sodic mentioned this pull request May 22, 2025

Refactor deploy package error handling #2782

Open

infomiho mentioned this pull request Jun 10, 2025

Create e2e test that tries to run Dockerfile Wasp creates via wasp build #884

Closed

		if (exitCode !== 0) {
		log("smtp-server", "error", `SMTP server exited with code ${exitCode}`);

Uh oh!

Run headless tests on todoApp production build #2707

Run headless tests on todoApp production build #2707

Uh oh!

Conversation

infomiho commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

FranjoMindek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

infomiho May 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FranjoMindek left a comment

Choose a reason for hiding this comment

Uh oh!

sodic left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sodic May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FranjoMindek May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Run headless tests on `todoApp` production build #2707

Run headless tests on `todoApp` production build #2707

infomiho commented Apr 28, 2025 •

edited

Loading

infomiho May 2, 2025 •

edited

Loading

sodic left a comment •

edited

Loading

sodic May 6, 2025 •

edited

Loading

FranjoMindek May 6, 2025 •

edited

Loading