While it's a bit niche, worth mentioning how Microsoft's other tool, Playwright (the E2E test runner w/ browser automation) handles the challenge of merging reports from multiple parallel tests.

[jdisco86]

          While it's a bit niche, worth mentioning how Microsoft's other tool, Playwright (the E2E test runner w/ browser automation) handles the challenge of merging reports from multiple parallel tests.

Since they have native support for sharding (scaling parallel test execution by running tests on multiple machines simultaneously), they've shared a GH Actions example which leverages actions/upload-artifact and actions/download-artifact as follows:

jobs:
  run-tests:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2, 3, 4]
        shardTotal: [4]

    steps:
    - name: Run Playwright tests
      run: npx playwright test --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}

    - name: Upload blob report to GitHub Actions Artifacts
      if: always()
      uses: actions/upload-artifact@v4
      with:
        name: blob-report-${{ matrix.shardIndex }}
        path: blob-report
        retention-days: 1

  merge-reports:
    # Merge reports after run-tests, even if some shards have failed
    if: always()
    needs: [run-tests]
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with:
        node-version: 18
    - name: Install dependencies
      run: npm ci

    - name: Download blob reports from GitHub Actions Artifacts
      uses: actions/download-artifact@v4
      with:
        path: all-blob-reports
        pattern: blob-report-*
        merge-multiple: true

    - name: Merge into HTML Report
      run: npx playwright merge-reports --reporter html ./all-blob-reports 

    - name: Upload HTML report
      uses: actions/upload-artifact@v4
      with:
        name: html-report--attempt-${{ github.run_attempt }}
        path: playwright-report
        retention-days: 14

While the added HTML-report related steps are specific for Playwright's use-case, the merge-multiple: true line is the crux here for pulling together the partial blob reports together, as required.

It's not exactly straightforward by a long-shot, but it is an "officially" recommended method for merging outputs together cohesively from prior jobs running matrix strategy.

Originally posted by @rdhar in actions/runner#2477 (comment)

[subir0071]

Hi @jdisco86 - Thanks for bringing this issue to our notice. Will check the same and provide our update.

[subir0071]

Hi @jdisco86 - This issue does not seem to be related to the runner image. Rather looks more to be a playwright issue. Could you please elaborate on the error mesage?