cmd/go: go 1.22.0 inconsistent and variable code coverage output for the same folder

[dev-gto]

Go version

go version go1.22.0 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/hamada/.cache/go-build'
GOENV='/home/hamada/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/hamada/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/hamada/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go1.22.0'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go1.22.0/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/hamada/go/src/gitlab.in.registro.br/dev/br/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3637209596=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Having a directory called, e.g., holiday with some go files and their corresponding test files, I called the go test -cover from the parent directory. Depending on directories arguments, I have different percentage values, which wasn't occurring on go < 1.22.0.

What did you see happen?

DEV=dev go test -tags dev -cover -buildvcs=false -ldflags '-s -w' ./holiday/
ok  	gitlab.in.registro.br/dev/br/core/holiday	(cached)	coverage: 97.6% of statements

DEV=dev go test -tags dev -cover -buildvcs=false -ldflags '-s -w' ./h*
...
ok  	gitlab.in.registro.br/dev/br/core/holiday	0.005s	coverage: 47.3% of statements

DEV=dev go test -tags dev -cover -buildvcs=false -ldflags '-s -w' ./...
...
ok  	gitlab.in.registro.br/dev/br/core/holiday	(cached)	coverage: 23.5% of statements
...

What did you expect to see?

A consistent percentage value (in this case, 97.6%) for all commands:

DEV=dev go test -tags dev -cover -buildvcs=false -ldflags '-s -w' ./holiday/
ok  	gitlab.in.registro.br/dev/br/core/holiday	(cached)	coverage: 97.6% of statements

DEV=dev go test -tags dev -cover -buildvcs=false -ldflags '-s -w' ./h*
...
ok  	gitlab.in.registro.br/dev/br/core/holiday	0.005s	coverage: 97.6% of statements

DEV=dev go test -tags dev -cover -buildvcs=false -ldflags '-s -w' ./...
...
ok  	gitlab.in.registro.br/dev/br/core/holiday	(cached)	coverage: 97.6% of statements
...

[bcmills]

This is almost certainly an effect of recent changes in -coverpkg:

• cmd/cover: regression in 1.20, cover tool no longer includes packages without tests #58770 (Go 1.22.0)
• cmd/cover: extend coverage testing to include applications #51430 (Go 1.19)

See previously:

• cmd/go: -coverpkg=all gives different coverage value when run on a package list vs ./... #23883

(attn @thanm)

[thanm]

Thanks for the report. I'll take a look.

[thanm]

@dev-gto would you kindly post a pointer to the code that you are working with, so that I can reproduce the problem on my machine? Thanks.

[dev-gto]

@thanm , actually my project is a private and large one.

Despite some testing errors (the original project does not emit any errors), you can reproduce with
https://github.com/registrobr/zxcvbn-go

go test -cover ./...
ok  	github.com/registrobr/zxcvbn-go	(cached)	coverage: 60.6% of statements
ok  	github.com/registrobr/zxcvbn-go/adjacency	(cached)	coverage: 40.8% of statements
	github.com/registrobr/zxcvbn-go/data		coverage: 0.0% of statements
	github.com/registrobr/zxcvbn-go/data/ptbr		coverage: 0.0% of statements
ok  	github.com/registrobr/zxcvbn-go/entropy	(cached)	coverage: 35.9% of statements
...

go test -cover ./[ae]*
ok  	github.com/registrobr/zxcvbn-go/adjacency	(cached)	coverage: 83.3% of statements
ok  	github.com/registrobr/zxcvbn-go/entropy	(cached)	coverage: 50.0% of statements

[AlexanderYastrebov]

#62212 might be related

[aaron42net]

GOEXPERIMENT=nocoverageredesign go test -cover ./... does appear to provide the expected coverage result, matching running go test -cover for each package separately. So this does seem related to the coverage redesign.

I've confirmed this with both @dev-gto's example above and a large private monorepo where I also encountered the issue.

[dev-gto]

Indeed, GOEXPERIMENT=nocoverageredesign returns the expected values on my original files