Update for moving rootfs images to GitHub releases in boot-utils #577
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Many of the paths in continuous-integration2 are relative, which can make it difficult to verify that files are placed in the correct place. To make this a little more ergonomic to work with, switch exclusively to pathlib and use absolute paths to all files. This allows us to deduplicate and simplify the code in a few places using pathlib's methods, which are a little higher level than the os functions used in a few places. Signed-off-by: Nathan Chancellor <[email protected]>
Signed-off-by: Nathan Chancellor <[email protected]>
This will prevent us from hitting GitHub API rate limits when fetching rootfs images within boot-utils, as we will cache the release JSON as a file during the TuxSuite step and pass it along to all the boot jobs like we do builds.json. The curl command is simple enough that it could be included in generate_workflow.py directly but PyYAML makes printing it somewhat complicated because of the quotations that are needed around the headers. Having a separate script is less brittle for a few extra lines of code. Signed-off-by: Nathan Chancellor <[email protected]>
Now that the rootfs images are not included in the boot-utils repository, we can just download the Python files from GitHub directly, which both removes the need to keep this up to date manually and will make getting boots under way quicker, as the repo history with all the rootfs images will not need to be cloned first. Signed-off-by: Nathan Chancellor <[email protected]>
Signed-off-by: Nathan Chancellor <[email protected]>
nathanchance
force-pushed
the
boot-utils-update
branch
from
913c64e to
3c2d56b
Compare
We only need boot-qemu.py or boot-uml.py per job, not both, so only download which one is needed for the current job. Additionally, use the boot_utils variable in more places and clean up the number of variables in the boot-utils download block. Signed-off-by: Nathan Chancellor <[email protected]>
nickdesaulniers
approved these changes
Jun 9, 2023
[skip actions] Signed-off-by: Nathan Chancellor <[email protected]>
|
I have kicked off a presubmit run, might take some time for it to finish, I will address any issues that it finds then merge this. https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/5225765696 |
|
Doesn't this need to be merged first before the run, to see if this PR breaks anything? |
|
I pushed my branch to this repo as |
I failed to notice that the "dtbs" folder is needed in the URL too, which breaks downloading the dtb. Pull the common path into its own variable so that it is more obvious how this is expected to work. Signed-off-by: Nathan Chancellor <[email protected]>
|
85c2476 deals with the only issue presubmit sees, so I am merging this. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request is basically the second part of ClangBuiltLinux/boot-utils#105.
The first change is just a clean up/update to consistently use absolute paths and pathlib, which is a bit more modern than the os methods we currently use and helps simplify things a bit in various places. This helps ensure everything is put in the correct place.
The second change downloads boot-utils straight from GitHub, which will allow us to drop the submodule check in, which will ultimately speed up the clones. After the aforementioned boot-utils pull request, the
*.pyfiles just need to reside in the same folder together, as they will automatically download the rootfs from GitHub.The third change helps us avoid a theoretical rate limit from querying GitHub's API for the latest boot-utils images release. Rather than querying the API dynamically in every job that will be using boot-utils, we query it in the TuxSuite job, cache it via upload artifact, and pull it down in the child job, so the API will only be queried three times per workflow.
Finally, the final two changes remove the submodule and updates the workflows with the new steps.
See the commit messages for full details.