[ci] Airgapped builds failing

[dmcardle]

FYI: @timothytrippel @milesdai

I'm seeing "Test an airgapped Bazel build" fail on an unrelated PR (#16277).

https://dev.azure.com/lowrisc/opentitan/_build/results?buildId=106356&view=logs&j=62754bfd-1e07-590e-4563-63d5dff44c03&t=da39aa2e-f818-5d8e-57ef-95af7701b447&l=133

ERROR: An error occurred during the fetch of repository 'lowrisc_misc_linters_wheels':
   Traceback (most recent call last):
	File "/home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/lowrisc_lint/rules/pip.bzl", line 49, column 13, in _pip_wheel_impl
		fail("pip_wheel failed: {} ({})".format(result.stdout, result.stderr))
Error in fail: pip_wheel failed: Defaulting to user installation because normal site-packages is not writeable
Collecting wheel
  Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel
 (ERROR: Could not install packages due to an OSError: [Errno 17] File exists: '/home/vsts/.local/lib/python3.9/site-packages'

WARNING: You are using pip version 22.0.3; however, version 22.3.1 is available.
You should consider upgrading via the '/home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/python3_x86_64-unknown-linux-gnu/bin/python3 -m pip install --upgrade pip' command.
)
ERROR: /home/vsts/work/1/s/WORKSPACE:35:10: fetching pip_wheel rule //external:lowrisc_misc_linters_wheels: Traceback (most recent call last):
	File "/home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/lowrisc_lint/rules/pip.bzl", line 49, column 13, in _pip_wheel_impl
		fail("pip_wheel failed: {} ({})".format(result.stdout, result.stderr))
Error in fail: pip_wheel failed: Defaulting to user installation because normal site-packages is not writeable
Collecting wheel
  Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel
 (ERROR: Could not install packages due to an OSError: [Errno 17] File exists: '/home/vsts/.local/lib/python3.9/site-packages'

WARNING: You are using pip version 22.0.3; however, version 22.3.1 is available.
You should consider upgrading via the '/home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/python3_x86_64-unknown-linux-gnu/bin/python3 -m pip install --upgrade pip' command.
)
INFO: Repository bindgen_clang_linux instantiated at:
  /home/vsts/work/1/s/WORKSPACE:56:10: in <toplevel>
  /home/vsts/work/1/s/third_party/rust/deps.bzl:36:30: in rust_deps
  /home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/rules_rust/bindgen/repositories.bzl:26:32: in rust_bindgen_dependencies
  /home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/rules_rust/bindgen/repositories.bzl:97:10: in _bindgen_clang_repositories
  /home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/bazel_tools/tools/build_defs/repo/utils.bzl:233:18: in maybe
Repository rule http_archive defined at:
  /home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in <toplevel>
ERROR: error loading package 'quality': at /home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/lowrisc_lint/rules/rules.bzl:8:6: Encountered error while reading extension file 'requirements.bzl': no such package '@lowrisc_misc_linters_pip//': no such package '@lowrisc_misc_linters_wheels//': pip_wheel failed: Defaulting to user installation because normal site-packages is not writeable
Collecting wheel
  Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Installing collected packages: wheel
 (ERROR: Could not install packages due to an OSError: [Errno 17] File exists: '/home/vsts/.local/lib/python3.9/site-packages'

WARNING: You are using pip version 22.0.3; however, version 22.3.1 is available.
You should consider upgrading via the '/home/vsts/.cache/bazel/_bazel_vsts/5c27df7f718ab35828b0e0cfe0993663/external/python3_x86_64-unknown-linux-gnu/bin/python3 -m pip install --upgrade pip' command.
)

[timothytrippel]

IIRC, #15176 tried to fix this, but it doesn't seem it succeeded.... darn.

[sha-ron]

Did the airgapped build passed before? I am trying it also on older and it fails but differently:

Analyzing: target //sw/device/tests:pwrmgr_smoketest_sim_dv (2 packages loaded, 0 targets configured)
ERROR: opentitan/WORKSPACE:43:9: fetching pip_wheel rule //external:ot_python_wheels: com.google.devtools.build.lib.vfs.FileAccessException: /home/users/stopaz/_bazel_stopaz/aebc30c563de29491e41950212af966b/external/ot_python_wheels/WORKSPACE (Permission denied)
ERROR: opentitan/sw/device/tests/BUILD:1552:19: error loading package 'util': Encountered error while reading extension file 'requirements.bzl': no such package '@ot_python_deps//': no such package '@ot_python_wheels//': /home/users/stopaz/_bazel_stopaz/aebc30c563de29491e41950212af966b/external/ot_python_wheels/WORKSPACE (Permission denied) and referenced by '//sw/device/tests:pwrmgr_smoketest_sim_dv'

[timothytrippel]

@sha-ron The issue you describe above ^ is different than the issue described in this issue. The error described in this issue is a transient issue. The airgapped build passes in CI currently I believe (it only fails intermittently).

[msfschaffner]

I don't think this is failing anymore - otherwise please reopen the issue.