Bazel tests are flaky

[johanbrandhorst]

The bazel tests fail with connection refused errors every now and then. Seems we might need to wait a little longer for the test server to start.

[johanbrandhorst]

Happened again today, but seems to be bazel consuming too many resources and the host killing the process? Not quite sure.

https://circleci.com/gh/grpc-ecosystem/grpc-gateway/3484

[achew22]

My vote is that we activate the Bazel team's CI and deactivate the CircleCI based one. Step 1 in that process is here #983

[srenatus]

Cool. So, in buildkite.com/bazel, test fail consistently. Like this one: https://buildkite.com/bazel/grpc-ecosystem-grpc-gateway/builds/104#8f57d1c1-3082-4a3d-8ad3-f1c35f0bf987

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //examples/integration:go_default_test
-----------------------------------------------------------------------------
E0911 09:09:51.099595      14 main.go:73] Failed to listen and serve: listen tcp :8080: bind: address already in use
cannot run gateway service: listen tcp :8080: bind: address already in use

(from the uploaded test logs in buildkite artifacts)

👀 I guess it's either running test in parallel that shouldn't, or... something else is bound to 8080. Either way, making the tests use a random available port (i.e. binding to :0, then retrieving the port that was used) could be an alternative 🤔

[johanbrandhorst]

TIL. Sounds like unintentional parallelism to me. Unfortunately I understand nothing of how this works, so I defer to @achew22.

[srenatus]

Well, I'm sure our tests would be more robust if they could run in parallel. 😃 But yeah, that's the harder route for sure.

[drigz]

FYI, if you add tags = ["exclusive"] for a Bazel test it will not be run in parallel with other tests, so you can use this for tests that bind to a fixed port.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[johanbrandhorst]

Flipping stale bot giving me lip

[johanbrandhorst]

Still flaky: https://circleci.com/gh/grpc-ecosystem/grpc-gateway/5222

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.