Skip to content

Reporting subtests as they finish #4250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Feb 8, 2025
Merged

Reporting subtests as they finish #4250

merged 5 commits into from
Feb 8, 2025

Conversation

linzhp
Copy link
Contributor

@linzhp linzhp commented Feb 7, 2025 •
edited
Loading

What type of PR is this?
Feature

What does this PR do? Why is it needed?
This PR makes rules_go report all finished subtests in the XML file, even when the test target times out. It does this by adding -test.v=test2json when verbose mode is requested. According to Go's comment:

If test2json is used, we never flush to parent tests, so that the json stream shows subtests as they finish.

It also sync test2json.go from upstream golang/go.

With a test file like this:

package test_report

import (
	"fmt"
	"testing"
	"time"
)

func TestReport(t *testing.T) {
	for i := 0; i < 10; i++ {
		t.Run(fmt.Sprintf("test %d", i), func(t *testing.T) {
			time.Sleep(2 * time.Second)
		})
	}
}

Running with --test_timeout=10 --test_env=GO_TEST_WRAP_TESTV=1, the XML before the PR:

<testsuites>
	<testsuite errors="6" failures="0" skipped="0" tests="6" time="" name="example.com/test_report/go_default_test">
		<testcase classname="go_default_test" name="TestReport" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport&#xA;</error>
		</testcase>
		<testcase classname="go_default_test" name="TestReport/test_0" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport/test_0&#xA;</error>
		</testcase>
		<testcase classname="go_default_test" name="TestReport/test_1" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport/test_1&#xA;</error>
		</testcase>
		<testcase classname="go_default_test" name="TestReport/test_2" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport/test_2&#xA;</error>
		</testcase>
		<testcase classname="go_default_test" name="TestReport/test_3" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport/test_3&#xA;</error>
		</testcase>
		<testcase classname="go_default_test" name="TestReport/test_4" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport/test_4&#xA;panic: test timed out after 10s&#xA;&#x9;running tests:&#xA;&#x9;&#x9;TestReport (10s)&#xA;&#x9;&#x9;TestReport/test_4 (2s)&#xA;&#xA;goroutine 22 [running]:&#xA;testing.(*M).startAlarm.func1()&#xA;&#x9;GOROOT/src/testing/testing.go:2373 +0x385&#xA;created by time.goFunc&#xA;&#x9;GOROOT/src/time/sleep.go:215 +0x2d&#xA;&#xA;goroutine 1 [chan receive]:&#xA;testing.(*T).Run(0xc0001aa000, {0x5b8d09?, 0x0?}, 0x5db1c0)&#xA;&#x9;GOROOT/src/testing/testing.go:1751 +0x3ab&#xA;testing.runTests.func1(0xc0001aa000)&#xA;&#x9;GOROOT/src/testing/testing.go:2168 +0x37&#xA;testing.tRunner(0xc0001aa000, 0xc00015cb18)&#xA;&#x9;GOROOT/src/testing/testing.go:1690 +0xf4&#xA;testing.runTests(0xc0001a2000, {0x712a50, 0x1, 0x1}, {0xc00015cb48?, 0x16?, 0x721b00?})&#xA;&#x9;GOROOT/src/testing/testing.go:2166 +0x43d&#xA;testing.(*M).Run(0xc0001521e0)&#xA;&#x9;GOROOT/src/testing/testing.go:2034 +0x64a&#xA;go.uber.org/goleak.VerifyTestMain({0x60dc20?, 0xc0001521e0?}, {0x0, 0x0, 0x0})&#xA;&#x9;external/org_uber_go_goleak/testmain.go:53 +0x5a&#xA;main.main()&#xA;&#x9;bazel-out/k8-fastbuild/bin/src/code.uber.internal/devexp/test_report/go_default_test_/testmain.go:130 +0x355&#xA;&#xA;goroutine 17 [syscall]:&#xA;os/signal.signal_recv()&#xA;&#x9;GOROOT/src/runtime/sigqueue.go:152 +0x29&#xA;os/signal.loop()&#xA;&#x9;GOROOT/src/os/signal/signal_unix.go:23 +0x13&#xA;created by os/signal.Notify.func1.1 in goroutine 1&#xA;&#x9;GOROOT/src/os/signal/signal.go:151 +0x1f&#xA;&#xA;goroutine 18 [chan receive]:&#xA;testing.(*T).Run(0xc0001aa1a0, {0xc000282060?, 0xc00015cf50?}, 0x5db310)&#xA;&#x9;GOROOT/src/testing/testing.go:1751 +0x3ab&#xA;src/code.uber.internal/devexp/test_report/go_default_test.TestReport(0xc0001aa1a0)&#xA;&#x9;src/code.uber.internal/devexp/test_report/a_test.go:11 +0x74&#xA;testing.tRunner(0xc0001aa1a0, 0x5db1c0)&#xA;&#x9;GOROOT/src/testing/testing.go:1690 +0xf4&#xA;created by testing.(*T).Run in goroutine 1&#xA;&#x9;GOROOT/src/testing/testing.go:1743 +0x390&#xA;&#xA;goroutine 34 [sleep]:&#xA;time.Sleep(0x77359400)&#xA;&#x9;GOROOT/src/runtime/time.go:315 +0xf2&#xA;src/code.uber.internal/devexp/test_report/go_default_test.TestReport.func1(0xc0002961a0?)&#xA;&#x9;src/code.uber.internal/devexp/test_report/a_test.go:12 +0x18&#xA;testing.tRunner(0xc0002961a0, 0x5db310)&#xA;&#x9;GOROOT/src/testing/testing.go:1690 +0xf4&#xA;created by testing.(*T).Run in goroutine 18&#xA;&#x9;GOROOT/src/testing/testing.go:1743 +0x390&#xA;</error>
		</testcase>
	</testsuite>
</testsuites>

The XML after the PR:

<testsuites>
	<testsuite errors="2" failures="0" skipped="0" tests="6" time="" name="example.com/test_report/go_default_test">
		<testcase classname="go_default_test" name="TestReport" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport&#xA;</error>
		</testcase>
		<testcase classname="go_default_test" name="TestReport/test_0" time="2.000"></testcase>
		<testcase classname="go_default_test" name="TestReport/test_1" time="2.000"></testcase>
		<testcase classname="go_default_test" name="TestReport/test_2" time="2.000"></testcase>
		<testcase classname="go_default_test" name="TestReport/test_3" time="2.000"></testcase>
		<testcase classname="go_default_test" name="TestReport/test_4" time="">
			<error message="No pass/skip/fail event found for test" type="">=== RUN   TestReport/test_4&#xA;panic: test timed out after 10s&#xA;&#x9;running tests:&#xA;&#x9;&#x9;TestReport (10s)&#xA;&#x9;&#x9;TestReport/test_4 (2s)&#xA;&#xA;goroutine 21 [running]:&#xA;testing.(*M).startAlarm.func1()&#xA;&#x9;GOROOT/src/testing/testing.go:2373 +0x385&#xA;created by time.goFunc&#xA;&#x9;GOROOT/src/time/sleep.go:215 +0x2d&#xA;&#xA;goroutine 1 [chan receive]:&#xA;testing.(*T).Run(0xc0001aa000, {0x5b8d07?, 0x0?}, 0x5db1d0)&#xA;&#x9;GOROOT/src/testing/testing.go:1751 +0x3ab&#xA;testing.runTests.func1(0xc0001aa000)&#xA;&#x9;GOROOT/src/testing/testing.go:2168 +0x37&#xA;testing.tRunner(0xc0001aa000, 0xc00015cb18)&#xA;&#x9;GOROOT/src/testing/testing.go:1690 +0xf4&#xA;testing.runTests(0xc0001a2000, {0x712a50, 0x1, 0x1}, {0xc00015cb48?, 0x16?, 0x721b00?})&#xA;&#x9;GOROOT/src/testing/testing.go:2166 +0x43d&#xA;testing.(*M).Run(0xc0001521e0)&#xA;&#x9;GOROOT/src/testing/testing.go:2034 +0x64a&#xA;go.uber.org/goleak.VerifyTestMain({0x60dc40?, 0xc0001521e0?}, {0x0, 0x0, 0x0})&#xA;&#x9;external/org_uber_go_goleak/testmain.go:53 +0x5a&#xA;main.main()&#xA;&#x9;bazel-out/k8-fastbuild/bin/src/code.uber.internal/devexp/test_report/go_default_test_/testmain.go:130 +0x355&#xA;&#xA;goroutine 17 [syscall]:&#xA;os/signal.signal_recv()&#xA;&#x9;GOROOT/src/runtime/sigqueue.go:152 +0x29&#xA;os/signal.loop()&#xA;&#x9;GOROOT/src/os/signal/signal_unix.go:23 +0x13&#xA;created by os/signal.Notify.func1.1 in goroutine 1&#xA;&#x9;GOROOT/src/os/signal/signal.go:151 +0x1f&#xA;&#xA;goroutine 18 [chan receive]:&#xA;testing.(*T).Run(0xc0001aa1a0, {0xc000282086?, 0xc00015cf50?}, 0x5db320)&#xA;&#x9;GOROOT/src/testing/testing.go:1751 +0x3ab&#xA;src/code.uber.internal/devexp/test_report/go_default_test.TestReport(0xc0001aa1a0)&#xA;&#x9;src/code.uber.internal/devexp/test_report/a_test.go:11 +0x74&#xA;testing.tRunner(0xc0001aa1a0, 0x5db1d0)&#xA;&#x9;GOROOT/src/testing/testing.go:1690 +0xf4&#xA;created by testing.(*T).Run in goroutine 1&#xA;&#x9;GOROOT/src/testing/testing.go:1743 +0x390&#xA;&#xA;goroutine 35 [sleep]:&#xA;time.Sleep(0x77359400)&#xA;&#x9;GOROOT/src/runtime/time.go:315 +0xf2&#xA;src/code.uber.internal/devexp/test_report/go_default_test.TestReport.func1(0xc000294340?)&#xA;&#x9;src/code.uber.internal/devexp/test_report/a_test.go:12 +0x18&#xA;testing.tRunner(0xc000294340, 0x5db320)&#xA;&#x9;GOROOT/src/testing/testing.go:1690 +0xf4&#xA;created by testing.(*T).Run in goroutine 18&#xA;&#x9;GOROOT/src/testing/testing.go:1743 +0x390&#xA;</error>
		</testcase>
	</testsuite>
</testsuites>

@linzhp linzhp requested a review from jayconrod February 7, 2025 06:10
fmeum
fmeum reviewed Feb 7, 2025
View reviewed changes
jayconrod
jayconrod approved these changes Feb 7, 2025
View reviewed changes
@xytan0056 xytan0056 mentioned this pull request Feb 7, 2025
Closed
@linzhp linzhp enabled auto-merge (squash) February 7, 2025 23:42
@linzhp linzhp merged commit 212b76d into bazel-contrib:master Feb 8, 2025
1 check passed
@linzhp linzhp deleted the test2json branch February 8, 2025 00:35
@linzhp linzhp mentioned this pull request Feb 11, 2025
Merged
fmeum pushed a commit that referenced this pull request Feb 11, 2025
@linzhp
Sync test2json from upstream Go (#4261)
4d0ddb7 
Sync test2json from upstream Go. All the code changes are from
golang/go. Only the comment was changed by this PR.

I tried to do that in #4250, but was blocked by a flaky test that was
fixed in #4257. Retrying...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jayconrod jayconrod jayconrod approved these changes

@fmeum fmeum fmeum approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@linzhp @jayconrod @fmeum