PyTorch CI Stability Action Tracker

[janeyx99]

This will be a meta-issue/tracker for directly actionable issues regarding stability and reliability in PyTorch CI.

Motivation

CI stability is crucial to developer velocity and user experience. Otherwise, developers may waste time debugging their CI failures that are irrelevant to their changes and the on-call who is in charge of keeping trunk green may be tasked with more work to triage issues. When trunk is red, PyTorch users who pull from source could be impacted as well.

Immediately Actionable Issues

• Modify Dr. CI so it could detect runner disconnection failures #66902
• [CI] Flaky initialization of OMP #79055
• android-tests is often flaky #79785
• Bazel build is very flaky #78100

Windows Specific

none so far!

Rolled into bigger projects (please comment relevant suggestions in their respective issues)

• Flaky Test Detection + Retry Automatically Detect and Remove Flaky Tests #71005
• Removing Network Connectivity Risks In CI CI: Removing Network Connectivity Risks #71003
• Retryable Steps GHA: Use retryable steps #71563

[suo]

Using pretrained weights from torchvision causes a network call to download weights. Example: https://github.com/pytorch/pytorch/blob/master/test/onnx/test_pytorch_onnx_onnxruntime.py#L354

Found from failure: https://circleci.com/gh/pytorch/pytorch/16139241

A cursory search shows quite a few tests where we are doing this.

Fixing is probably as easy as just switching pretrained to False—I highly doubt any of those tests are sensitive to whether the weights are pretrained are not.

[suo]

Looks like CMake binary is not available?

Example: https://github.com/pytorch/pytorch/runs/3677706168

I'm going to chalk this one up as "probably fixed by ephemeral runners"

[suo]

Downloading nvidia stuffs in test failed:
https://github.com/pytorch/pytorch/runs/3681335684

Would be resolved by baking into the AMI

[suo]

sccache on windows has had network failures 25 times in the last week: https://fburl.com/scuba/opensource_ci_jobs/q3nf3g6n

Googling the error message shows an issue that @ezyang commented on: mozilla/sccache#256. Suggested fixes in that issue are 1) reducing parallelism, and 2) increasing the timeout

[janeyx99]

Problems noted by @mruberry:

Run rm -rf "${PYTORCH_FINAL_PACKAGE_DIR}"
rm: cannot remove './build': Device or resource busy

in https://github.com/pytorch/pytorch/runs/3682204176 and

Run seemethere/upload-artifact-s3@v3
Error: No files were found with the provided path: C:\1264644579\build-results. No artifacts will be uploaded.

in https://github.com/pytorch/pytorch/runs/3683674360 and

Run docker pull "${DOCKER_IMAGE}"
Error response from daemon: Get "https://308535385114.dkr.ecr.us-east-1.amazonaws.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

in https://github.com/pytorch/pytorch/runs/3682134416 and

Sep 23 02:20:31   test_error_in_init (__main__.TestDataLoader) ... ok (0.164s)

Too long with no output (exceeded 1h30m0s): context deadline exceeded

in https://app.circleci.com/pipelines/github/pytorch/pytorch/384098/workflows/c9f78d92-487c-4a85-b97e-10b222e5c3a9/jobs/16158997 (an ASAN test job).

[suo]

Run seemethere/upload-artifact-s3@v3
Error: No files were found with the provided path: C:\1264644579\build-results. No artifacts will be uploaded.

This is not a real failure, it's a build failure. We should mark this step as "continue-through-error" or whatever because it's not a user-facing failure.

[driazati]

Run seemethere/upload-artifact-s3@v3
Error: No files were found with the provided path: C:\1264644579\build-results. No artifacts will be uploaded.

This is not a real failure, it's a build failure. We should mark this step as "continue-through-error" or whatever because it's not a user-facing failure.

Still something we can fix though, we should have guards on those steps instead of if: always(), they should check if the build step succeeded

[janeyx99]

Still something we can fix though, we should have guards on those steps instead of if: always(), they should check if the build step succeeded

I'll take this--I've modified the big summary at the top to include this action item and assigned it to me

[suo]

Raft of issues from spot check today:

• mt.exe manifest update failure: https://github.com/pytorch/pytorch/runs/3806258421 I have no idea wtf this means, but this SO answer seemed relevant https://stackoverflow.com/a/42136824
• network failure: https://github.com/pytorch/pytorch/runs/3798462187
• network failure: https://circleci.com/gh/pytorch/pytorch/16331602
• flaky test: https://github.com/pytorch/pytorch/runs/3804782432 tracked in DISABLED test_profiler (__main__.TestJit) #65521
• flaky test: https://circleci.com/gh/pytorch/pytorch/16332387, I alerted the relevant team

[suo]

PyTorch docs builds failing on installing dependencies: https://app.circleci.com/pipelines/github/pytorch/pytorch/393635/workflows/ba76eff9-428a-4e3d-8023-626b7413ed1f/jobs/16432372

Oct 13 06:43:37 ++ pip -q install -r requirements.txt
Oct 13 06:53:38   ERROR: Command errored out with exit status 128:
Oct 13 06:53:38    command: git clone -q https://github.com/pytorch/pytorch_sphinx_theme.git /var/lib/jenkins/workspace/docs/src/pytorch-sphinx-theme
Oct 13 06:53:38        cwd: None
Oct 13 06:53:38   Complete output (3 lines):
Oct 13 06:53:38   error: RPC failed; curl 18 transfer closed with outstanding read data remaining
Oct 13 06:53:38   fatal: The remote end hung up unexpectedly
Oct 13 06:53:38   fatal: protocol error: bad pack header

[suo]

Access denied on windows build: https://github.com/pytorch/pytorch/runs/3897754598 (cc @seemethere)

[suo]

Windows dependency installation failure: https://github.com/pytorch/pytorch/runs/3898324416?check_suite_focus=true

CondaError: Downloaded bytes did not match Content-Length
  url: https://repo.anaconda.com/pkgs/main/win-64/intel-openmp-2021.3.0-haa95532_3372.conda

Would be resolved by baking into AMI I guess

Here's another:
https://github.com/pytorch/pytorch/runs/3900764593

[abitrolly]

Is data about CI stability collected somewhere? Before making changes it would be nice to see and set some metrics.

https://observablehq.com/@observablehq/integration-test-flakiness

[suo]

This looks super cool! We do have data, but right now it's dumped to an internal system at FB (Meta??) for analysis. We'd love to make something that everyone can look at and help out with; I'll play with that notebook a bit.

For current master health, we have hud.pytorch.org, which presents a nice view of current jobs running on master.

[abitrolly]

@suo what is the process then to make the stats alive open data. Should I join Meta (FB ??) to champion the change? :D

[suo]

persistent dirty checkout, 27 failed jobs on master in the last week: https://fburl.com/scuba/opensource_ci_jobs/vs1j5mvq

[janeyx99]

Completed as of today (1.7.21):

• GHA: fully clean our workspace between runs and remove problematic non-ephemeral runners #69540 (thanks @malfet)!
• Ephemeral runners are available (thanks @seemethere)! However, because of capacity issues, they are currently not used everywhere.
• Tracked by pytorch_linux_xenial_py3_clang7_asan_test2 times out #65727: Investigate timeout in ASAN test job -- it looks like the runner gets stuck and outputs nothing for over 1.5hr. https://app.circleci.com/pipelines/github/pytorch/pytorch/384098/workflows/c9f78d92-487c-4a85-b97e-10b222e5c3a9/jobs/16158997 and https://app.circleci.com/pipelines/github/pytorch/pytorch/385616/workflows/2e3e950f-f3d3-4f05-a5fe-7f95fba436af/jobs/16214108
• Moved auto-label-rocm job to probot instead of as another GHA in pytorch/pytorch to reduce the number of GHA requests and prevent race conditions. Moreover, many of these workflows are cancelled due to concurrency limits and muddle our stats collection. - assigned to @janeyx99
• Ensure that the "Upload artifacts to S3" step will not fail as it shouldn't - assigned to @janeyx99 Windows GHA: Only upload artifacts if prev steps pass #65561
• Add retrying to "docker pull" .github: GHA retry docker pull #65103, that should solve problems such as https://github.com/pytorch/pytorch/runs/3683674360
• Improve retry for test cases that access the network: Catch http exceptions and retry them so we don't fail like in https://github.com/pytorch/pytorch/runs/3798462187. test_utils.py: Add another retry to test_download_url_to_file #66159
• Stop using pretrained weights from torchvision causes a network call to download weights. This has been done to its fullest in [BE] set pretrained=False for onnx tests #66312 and [skip ci] Change pretrained to false for quantization tests #66795, where the remaining instances seem to semantically matter.
• Baked CUDA installation and VS installation into the base AMI to remove network connectivity risk in CI.
• Rehosted bazel download on our S3 instead of GitHub ci: Move bazel download from github -> s3 #66815 to help fix bazel CI running into "403 Forbidden" when installing dependencies #66810

No longer relevant/De-prioritized

• Adding retries to Circle steps "docker pull" and "docker sign-in" Add retries to docker sign-in and docker pull for all CI #66508 no longer relevant as they've been migrated to GHA
  • Circle docker pulls: docker pull error on a PR https://app.circleci.com/pipelines/github/pytorch/pytorch/390580/workflows/7daa146f-5883-46b7-b567-7c305e6b92e9/jobs/16358554
  • failure in build-publish-docker here: https://github.com/pytorch/pytorch/runs/3815292363?check_suite_focus=true
• Add periodic cron to dispatch workflows for trunk commits with missing workflow triggers due to ShipIt batching - assigned to @janeyx99
• Since this failure has not re-occurred, there's no need to look into this right now: Chown workspace failing with ./.git/index.lock: No such file or directory in https://github.com/pytorch/pytorch/runs/3692140415

[janeyx99]

Completed between 1.7.21 to 4.11

• Add retries to ecr get-login call: https://github.com/pytorch/pytorch/runs/3948026530 #67699
• Windows CI "No module named 'torch'" #69730
• Windows build in CI failing due to mt.exe manifest issue #66278
• Alleviate sccache connectivity issues on CI #67720

[janeyx99]

Completed between 4.11 and 6.17

• Fix docker update race condition #70881

[ZainRizvi]

Closing the meta tracker since we aren't using it anymore