@stryker-mutator/tap-runner has unexpected interaction with the Node.js v24 build-in test runner

[ericcornelissen]

Summary

When upgrading from Node.js v22 to v24, the interaction between @stryker-mutator/tap-runner and Node's build-in test runner changed and now it reports n/a for tests with a 100% mutation score.

I have observed this behavior in two separate projects with (more or less) the same test setup:

• Bump Node.js version for development to v24 ericcornelissen/depreman#167
• https://gitlab.com/ericcornelissen/pp-test-kit/-/jobs/9999190125#L115

Stryker config

export default {
	coverageAnalysis: "perTest",
	inPlace: false,
	timeoutMS: 5000,

	mutate: [
		"src/**/*.js",
		"!src/**/*.test.js",
		"!src/main.js",
	],

	testRunner: "tap",
	tap: {
		testFiles: ["src/*.test.js"],
		forceBail: true,
	},

	incremental: true,
	incrementalFile: "node_modules/.cache/stryker-incremental.json",

	reporters: ["clear-text", "html", "progress"],
	htmlReporter: {
		fileName: "mutation.html",
	},

	thresholds: {
		high: 100,
		low: 96,
		break: 96,
	},

	tempDirName: "node_modules/.temp/stryker",
	cleanTempDir: true,
};

Test runner config

n/a

Stryker environment

├── @stryker-mutator/[email protected]
├── @stryker-mutator/[email protected]

Test runner environment

node --test 'src/*.test.js'

Your Environment

software	version(s)
node	24.0.0
npm	11.3.0
Operating System	Ubuntu 24.04

Add stryker.log

stryker.log

[nicojs]

Hi @ericcornelissen 👋

Hmm, interesting. The default --test-reporter changed in Node since node 23 from tap to spec. I can see it in the docs:
https://nodejs.org/api/test.html#test-reporters. It changed here: nodejs/node@eb7e18fe94, but I don't see it in the history table in the docs, so I just spent an hour second-guessing myself 😪

A simple workaround for now is to apply this diff:

diff --git a/.stryker.js b/.stryker.js
index 7c3386d..4683c98 100644
--- a/.stryker.js
+++ b/.stryker.js
@@ -14,6 +14,7 @@ export default {
        testRunner: "tap",
        tap: {
                testFiles: ["src/*.test.js"],
+               nodeArgs: ["-r", "{{hookFile}}", "--test-reporter", "tap", "{{testFile}}"],
                forceBail: true,
        },

We should document this and maybe provide --test-reporter tap as default.

[ericcornelissen]

Thanks you for figuring that out @nicojs 🙇

[nicojs]

Upgrading to StrykerJS 9 should fix it without you having to change your config file :)