go/*,cmd/gofmt: guard support for generic code behind a "typeparams" build constraint

[findleyr]

Our initial approach to hiding the type parameter API in go/* libraries merged from dev.typeparams was to guard things behind the go1.18 build constraint. Unfortunately, this makes it difficult to build and test a toolchain that supports type parameters.

As suggested by @bcmills in chat, using GOEXPERIMENT=typeparams satisfies our requirements:

• As of https://golang.org/cl/222925, it automatically sets the goexperiment.typeparams build constraint.
• It's straightforward to set up a builder to run with an experiment (as in x/build: create builder that runs with GOEXPERIMENT set to 'staticlockranking' (once lock ranking change is in) #37937).

I've integrated GOEXPERIMENT=typeparams in https://golang.org/cl/300649, however I'm not sure if I'm holding it right. The location of the GOEXPERIMENT definition (in cmd/internal/objabi) as well as its historical use does not suggest that it's a generally available knob. It's also a bit confusing to have the GOEXPERIMENT=typeparams used for go/* libraries and -G=N used for the compiler, especially since GOEXPERIMENT has historically been used for the compiler.

All we really need is a build constraint and a builder that runs tests with that build constraint. We could probably do this without a GOEXPERIMENT value, but it seems silly to invent a new mechanism when GOEXPERIMENT exists.

CC @griesemer @mdempsky @dmitshur for discussion. Any opinions on using GOEXPERIMENT? Any other ideas?

[gopherbot]

Change https://golang.org/cl/300649 mentions this issue: go/*,cmd/gofmt: add a new "typeparams" GOEXPERIMENT

[ianlancetaylor]

See #42681, which was just accepted.

[mdempsky]

For purely standard library features, we've usually just given them new build tags; e.g., "faketime", "nethttpomithttp2", "x509omitbundledroots", "osusergo", "netgo", "math_big_pure_go", "compiler_bootstrap", "libfuzzer", ... I think it would be fine to just declare another one for hiding go/*'s typeparams support.

As you point out, GOEXPERIMENT currently is more of a toolchain knob to control what experimental compiler/runtime features should be used for compiling the target program. It would be a new use of it to instead control what experimental standard library features are available. However, I'm not opposed to doing that if it plays more nicely with the build infrastructure or something.

As for how typeparams support is enabled in cmd/compile, I'd be fine with changing that to use GOEXPERIMENT too. Though whether a Go program itself uses type params and whether it wants to use go/* to process other Go code that use type params are somewhat two orthogonal issues. Again, I don't have any preference on whether these are controlled by a single knob or independently with two different knobs.

[findleyr]

I'd be fine with simply using a build tag, but do think it's important to have a builder running tests with typeparams enabled. Due to how integrated type parameters are with the type checker, it's possible for seemingly unrelated changes to affect type parameter checking.

Looking through x/build, I don't think it would be easy to run tests with e.g. -tags=typeparams. It looks like we'd have to do some non-trivial plumbing through cmd/dist and x/build. For example, I tried running go install -tags=goexperiment.typeparams std before running go tool dist test "go_test:go/types", and dist rebuilds without the tag before testing.

So my questions are:

• Is my assessment of the effort required in x/build to test with an arbitrary build tag correct? (this is mostly addressed to the release team)
• Does using GOEXPERIMENT for non compiler/runtime related experiments set a dangerous precedent?
• Should we "officially" expand the scope of GOEXPERIMENT to include standard library experiments, now that build: move GOEXPERIMENT knob from make.bash to cmd/go #42681 is accepted?

My current feeling is that the quick-and-dirty solution of defining a fake experiment requires approximately no work, and is probably justified for type parameter library changes due to the nature and scope of work on generics. I'm concerned that the use of GOEXPERIMENT will be confusing, but hopefully we can mitigate that confusion with commentary/documentation. Changing cmd/compile to also use GOEXPERIMENT would reduce confusion, but IMO is not strictly necessary.

CC @golang/release

[mdempsky]

I agree there should be builder test coverage. But can't you just add another stanza like this in cmd/dist/test.go?

go/src/cmd/dist/test.go

Lines 468 to 475 in 0a65559

	t.tests = append(t.tests, distTest{
	name: "osusergo",
	heading: "os/user with tag osusergo",
	fn: func(dt *distTest) error {
	t.addCmd(dt, "src", t.goTest(), t.timeout(300), "-tags=osusergo", "os/user")
	return nil
	},
	})

[findleyr]

Thanks, that's a good idea -- I'm not that familiar with cmd/dist. I was thinking about this in terms of adding a single linux-amd64-typeparams builder, but this might be better.

Well, for now I'm going to update my CL to just use a unique build constraint. We can independently solve the problem of how to test it.

[gopherbot]

Change https://golang.org/cl/310070 mentions this issue: go/ast: fix broken build with typeparams build constraint

[gopherbot]

Change https://golang.org/cl/310071 mentions this issue: cmd/dist: add tests using the typeparams build tag

[gopherbot]

Change https://golang.org/cl/317029 mentions this issue: go/types: expose more API under -tags=typeparams