[BUG] CLI mode sends flaky test details to thread while normal reporter mode doesn't

[miguelofbc]

Bug description

When using showInThread: true, the CLI mode incorrectly sends flaky test failure details to the thread, while the normal reporter mode correctly excludes them. This inconsistency causes unnecessary noise in Slack threads for tests that ultimately passed after retries.

Actual Behavior

• Normal mode: ✅ Only sends actual failures to thread (correct)
• CLI mode: ❌ Sends both failures AND flaky test details to thread (incorrect)

Root Cause
The issue is in src/ResultsParser.ts at line 121 in the parseTests() method:

// CLI mode uses global retry config for all tests
retries, // ← Global retry value from config.projects[0].retries

While normal mode uses per-test retry values:
// Normal mode gets actual per-test retry config
retries: testCase.retries, // ← Individual test retry configuration

This causes the failure detection logic in getFailures() to incorrectly classify flaky tests as failures in CLI mode:

// Line 187: This condition determines if a test should be sent to thread
if (test.retries === test.retry) {
// Only tests that failed on their FINAL retry should be here
}

Example Scenario
For a flaky test that fails on retry 0 but passes on retry 1:

Normal Mode (Correct):

• test.retry = 0 (failed attempt)
• test.retries = 1 (actual retry config)
• 0 !== 1 → Not sent to thread ✅

CLI Mode (Incorrect):

• test.retry = 0 (failed attempt)
• test.retries = 0 (global config value)
• 0 === 0 → Incorrectly sent to thread ❌

Solution
Replace the global retry configuration with actual retry information from the JSON data:

// In parseTests() method, lines 113-117
// Calculate actual retries based on the maximum retry attempt for this test
const maxRetryAttempt = test.results.length > 0
? Math.max(...test.results.map((r: any) => r.retry))
: 0;
const effectiveRetries = Math.max(maxRetryAttempt, retries);

// Use effectiveRetries instead of global retries
retries: effectiveRetries,

Benefits of This Solution:

1. Uses actual retry data from JSON instead of global config
2. Maintains backward compatibility by falling back to global config
3. Calculates effective retries based on what actually happened during test execution
4. Ensures consistency between CLI and normal reporter modes

How to reproduce (please include a snippet of your Playwright Slack Reporter configuration)

Expected behavior

• Only tests that failed on their final retry attempt should have their details sent to the thread
• Flaky tests (that passed after retries) should only appear in the summary statistics with 🟡 emoji
• CLI mode and normal reporter mode should behave consistently

Environment & setup

• OS: any (unix + macOS tested)
• Node.js version: 22.9

playwright-slack-report version

1.1.89

[miguelofbc]

@ryanrosello-og could you please take a look?