bazel coverage returns empty coverage files for cc_test with remote execution

[finn-ball]

Description of the problem:

Bazel coverage fails with remote execution for cc_test.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Either run bazel coverage on your own RE infrastructure of follow the example posted here:
#4685 (comment)

Using an example such as abseil-cpp:

bazel coverage --config=rbe //absl/flags:config_test  --experimental_split_coverage_postprocessing --experimental_fetch_all_coverage_outputs

You'll notice coverage executes but the coverage files are returned empty. These files are not empty if executed locally (without the extra flags or it will crash).

What operating system are you running Bazel on?

Linux 5.10.16-arch1-1

What's the output of bazel info release?

release 4.0.0

Have you found anything relevant by searching the web?

#13185
#4685

Any other information, logs, or outputs that you want to share?

[finn-ball]

@oquenchil
I suspect this line is not executing properly:

bazel/tools/test/collect_coverage.sh

Line 182 in 26c7e10

eval "${CC_CODE_COVERAGE_SCRIPT}"

When I look in the test logs, I see this:

cat bazel-out/k8-fastbuild/testlogs/absl/flags/config_test/test.log

external/bazel_tools/tools/test/collect_coverage.sh: line 182: external/bazel_tools/tools/test/collect_cc_coverage.sh: No such file or directory

Bazel should probably error out here.

[finn-ball]

This is what is causing empty files to be written:

bazel/tools/test/collect_coverage.sh

Line 176 in 26c7e10

touch $COVERAGE_OUTPUT_FILE

[finn-ball]

Merged cc_code_coverage.sh into collect_coverage.sh to unblock progress.

It looks like it is also failing on this line:

bazel/tools/test/collect_cc_coverage.sh

Line 129 in 26c7e10

cp "$ROOT/${gcno_path}" "${COVERAGE_DIR}/${gcno_path}"

The logs indicate it cannot find the files it wants which is interesting because I can find those files locally. They probably aren't being included

e.g

cp: cannot stat '/worker/build/752414a96e0dc08a/root/bazel-out/k8-fastbuild/bin/absl/flags/_objs/program_name/program_name.pic.gcno': No such file or directory

Yet I can see this file here:

bazel-abseil-cpp/bazel-out/k8-fastbuild/bin/absl/flags/_objs/program_name/program_name.pic.gcno

[finn-ball]

@oquenchil - Fairly sure the problem is that the .gcno files aren't added as a dependency to the part which needs to execute gcov. Looking through Bazel's coverage classes at the moment to try and figure out if I can add those files as a dependency.

[finn-ball]

I have managed to fix this by adding the following change to bazel 4.0.0.

diff --git a/src/main/java/com/google/devtools/build/lib/exec/StandaloneTestStrategy.java b/src/main/java/com/google/devtools/build/lib/exec/StandaloneTestStrategy.java
index fd1214d..3df29d2 100644
--- a/src/main/java/com/google/devtools/build/lib/exec/StandaloneTestStrategy.java
+++ b/src/main/java/com/google/devtools/build/lib/exec/StandaloneTestStrategy.java
@@ -452,6 +452,7 @@ public class StandaloneTestStrategy extends TestStrategy {
         action.getLcovMergerRunfilesSupplier(),
         /* filesetMappings= */ ImmutableMap.of(),
         /* inputs= */ NestedSetBuilder.<ActionInput>compileOrder()
+            .addAll(ImmutableList.copyOf(action.getInputs().toList()))
             .addAll(expandedCoverageDir)
             .add(action.getCollectCoverageScript())
             .add(action.getCoverageDirectoryTreeArtifact())

I'm curious as to why this is called StandaloneTestStrategy.java which it seems to be used in remotely sandboxed builds?

I figured out that the cc_code_coverage.sh and *.gcno files were missing from the CoveragePostProcessing spawn but were in the action, hence why I copied them over. Though there may be a better way of doing this. Thoughts?

[ulfjack]

It's called StandaloneTestStrategy for historical reasons. Google internally has different implementations for this depending on whether the test executes locally or remotely, but that isn't necessary in Bazel (and should also be fixed internally, but it's difficult because reasons). If it gets fixed at some point, this class will hopefully be merged into TestStrategy and go away.

The missing inputs certainly looks like an oversight. That should also break with sandboxing unless I'm mistaken, so it's odd this wasn't caught before. Are you already using the new code path?

[finn-ball]

That should also break with sandboxing unless I'm mistaken, so it's odd this wasn't caught before

These flags don't work locally with sandboxing: #13185 and my guess is that Bazel does not currently test this part of the codebase (at least in Bazel 4.0.0).

Interestingly if bazel/tools/test/collect_coverage.sh fails, Bazel doesn't currently catch this and will pass. If you check the test logs, you can see collect_coverage.sh cannot find collect_cc_coverage.sh (as it isn't included as an input) and will continue without erroring out. Since the file is touched (and thus created) right at the start, Bazel thinks it has worked.

Are you already using the new code path?

Yes as the workaround I've added finally allows you to easily return the coverage.dat files as non-empty with remote execution.

[finn-ball]

Fixed in :

ab1da7b

[nstng]

Hi, trying my luck here before opening a new issue:
We still see this behavior with Bazel 5.2.0 in CI on remote caches on cc_tests. bazel-out/_coverage/_coverage_report.dat is created when all tests are green. We have a couple of tests marked as flaky = True. Observation is that the coverage file is not created if one of these tests shows it's flaky behavior, i.e. fails in at least one run.
Any idea if this is related / is not covered by the fix?

[bhushan-helloFresh]

Hi All,
I am using Bazel 5.3.0 and the issue still persists. The coverage command is not failing but _coverage_report.dat file is always empty.
Any idea how to fix it?
@nstng @finn-ball

[finn-ball]

Can take a look if you have an example repo to share?

[bhushan-helloFresh]

Can't share the repo but its a monorepo written in Go and kotlin.

bazel coverage --combined_report=lcov //myapp/repo:all

myapp is the folder containing the repos and repo is the repo's name.
@finn-ball

[nstng]

Hi @bhushan-helloFresh ,
we had some good results creating the coverage with the additional flags --experimental_split_coverage_postprocessing --experimental_fetch_all_coverage_outputs --remote_download_outputs=all for the c tests. But for e.g., python tests the behavior is again different.
In our scenario coverage was sometimes not generated - it might be that your problem is something else as you said that your coverage is never generated.

[bhushan-helloFresh]

@nstng
Did you add any coverage-related changes in your BUILD.bazel and/or WORKSPACE?
I tried with additional flags, still empty!!

[wondfor]

what's the gcov versiion?

[bhushan-helloFresh]

@wondfor

Apple LLVM version 13.1.6 (clang-1316.0.21.2.5)
Optimized build.
Default target: arm64-apple-darwin21.5.0
Host CPU: vortex

[finn-ball]

I don't have a macbook infront of me today but could you quickly see if you can reproduce the problem with this repo?
https://github.com/abseil/abseil-cpp

[iminders]

• MacOS Monterey
• bazel 5.2.0
• Apple clang version 14.0.0 (clang-1400.0.29.202)

the same