Skip to content

ARM Megatests #1529

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 24 commits into
base: dev
Choose a base branch
from
Open

ARM Megatests #1529

wants to merge 24 commits into from
+3,206 −2,621

Conversation

edmundmiller
Copy link
Contributor

@edmundmiller edmundmiller commented Mar 26, 2025 •
edited
Loading

This adds ARM Megatests with an ARM-based compute environment I've set up.

@edmundmiller edmundmiller self-assigned this Mar 26, 2025
@edmundmiller edmundmiller requested review from ewels and mashehu March 26, 2025 14:36
@edmundmiller edmundmiller moved this to Waiting to be merged in arm64 software builds Mar 26, 2025
@edmundmiller edmundmiller added this to the 3.18.0 milestone Mar 26, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 26, 2025 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit be0cfc4

+| ✅ 191 tests passed       |+
#| ❔   9 tests were ignored |#
!| ❗  10 tests had warnings |!

❗ Test warnings:

  • files_exist - File not found: assets/multiqc_config.yml
  • files_exist - File not found: .github/workflows/awstest.yml
  • files_exist - File not found: .github/workflows/awsfulltest.yml
  • pipeline_todos - TODO string in nextflow.config: Update the field with the details of the contributors to your pipeline. New with Nextflow version 24.10.0
  • pipeline_todos - TODO string in ro-crate-metadata.json: "description": "

    \n \n <source media="(prefers-color-scheme: dark)" srcset="docs/images/nf-core-rnaseq_logo_dark.png">\n <img alt="nf-core/rnaseq" src="docs/images/nf-core-rnaseq_logo_light.png">\n \n

    \n\nGitHub Actions CI Status\nGitHub Actions Linting StatusAWS CICite with Zenodo\nnf-test\n\nNextflow\nrun with conda\nrun with docker\nrun with singularity\nLaunch on Seqera Platform\n\nGet help on SlackFollow on TwitterFollow on MastodonWatch on YouTube\n\n## Introduction\n\nnf-core/rnaseq is a bioinformatics pipeline that ...\n\n TODO nf-core:\n Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the\n major pipeline sections and the types of output it produces. You're giving an overview to someone new\n to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction\n\n\n Include a figure that guides the user through the major workflow steps. Many nf-core\n workflows use the "tube map" design for that. See https://nf-co.re/docs/contributing/design_guidelines#examples for examples. \n Fill in short bullet-pointed list of the default steps in the pipeline 1. Read QC (FastQC)2. Present QC for raw reads (MultiQC)\n\n## Usage\n\n> [!NOTE]\n> If you are new to Nextflow and nf-core, please refer to this page on how to set-up Nextflow. Make sure to test your setup with -profile test before running the workflow on actual data.\n\n Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.\n Explain what rows and columns represent. For instance (please edit as appropriate):\n\nFirst, prepare a samplesheet with your input data that looks as follows:\n\nsamplesheet.csv:\n\ncsv\nsample,fastq_1,fastq_2\nCONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz\n\n\nEach row represents a fastq file (single-end) or a pair of fastq files (paired end).\n\n\n\nNow, you can run the pipeline using:\n\n update the following command to include all required parameters for a minimal example \n\nbash\nnextflow run nf-core/rnaseq \\\n -profile <docker/singularity/.../institute> \\\n --input samplesheet.csv \\\n --outdir <OUTDIR>\n\n\n> [!WARNING]\n> Please provide pipeline parameters via the CLI or Nextflow -params-file option. Custom config files including those provided by the -c Nextflow option can be used to provide any configuration except for parameters; see docs.\n\nFor more details and further functionality, please refer to the usage documentation and the parameter documentation.\n\n## Pipeline output\n\nTo see the results of an example test run with a full size dataset refer to the results tab on the nf-core website pipeline page.\nFor more details about the output files and reports, please refer to the\noutput documentation.\n\n## Credits\n\nnf-core/rnaseq was originally written by Harshil Patel, Phil Ewels, Rickard Hammar\u00e9n.\n\nWe thank the following people for their extensive assistance in the development of this pipeline:\n\n If applicable, make list of people who have also contributed \n\n## Contributions and Support\n\nIf you would like to contribute to this pipeline, please see the contributing guidelines.\n\nFor further information or help, don't hesitate to get in touch on the Slack #rnaseq channel (you can join with this invite).\n\n## Citations\n\n Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. \n If you use nf-core/rnaseq for your analysis, please cite it using the following doi: 10.5281/zenodo.XXXXXX \n\n Add bibliography of tools and data used in your pipeline \n\nAn extensive list of references for the tools used by the pipeline can be found in the CITATIONS.md file.\n\nYou can cite the nf-core publication as follows:\n\n> The nf-core framework for community-curated bioinformatics pipelines.\n>\n> Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.\n>\n> Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.\n",
  • pipeline_todos - TODO string in base.config: Check the defaults for all processes
  • pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
  • pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
  • pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!
  • pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline

❔ Tests ignored:

✅ Tests passed:

Run details

  • nf-core/tools version 3.2.0
  • Run at 2025-05-09 17:34:08

@edmundmiller edmundmiller mentioned this pull request Mar 26, 2025
Closed
mashehu
mashehu reviewed Mar 27, 2025
View reviewed changes
@edmundmiller
ci: Update artifact name format in cloud tests workflow
738ddaa 
Changed the artifact name format for the Tower debug log to include aligner and architecture details for better traceability.

https://github.com/actions/upload-artifact/tree/v4/?tab=readme-ov-file#not-uploading-to-the-same-artifact
@nf-core-bot
Copy link
Member

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.2.0.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

@edmundmiller
refactor: Bring arm back, and just use arm on macos
be0cfc4 
Now that we have arm builds for most of the software
ewels
ewels reviewed May 14, 2025
View reviewed changes
nextflow.config
@@ -181,6 +181,7 @@ profiles {
apptainer.enabled = false
}
docker {
// includeConfig 'conf/containers/containers_docker_amd64.config'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include at the top, above profiles? That way it's the default for anyone not using profiles (eg. someone wanting to use Podman, or whatever).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we still need to solve image

@edmundmiller
Copy link
Contributor Author

@ewels I've been thinking a bit more about this.

I think we do this in phases.

Phase one, we start just generating the container configs and put them in the repo—nothing else. See if it anything breaks as we're generating them.

Phase two, we refactor the ARM profile. We make it a requirement for now to use wave if you're going to use arm, and we support the old methods of containers.

After we've figured out the stability of those, then we can move to the next phase of actually implementing this config module including.

@edmundmiller edmundmiller marked this pull request as ready for review May 15, 2025 13:48
@ewels
Copy link
Member

ewels commented May 16, 2025

You've done it manually here, right? I'm not sure what the problem is with including new configs / profiles that are manually generated if they use new profile names for now (_beta suffix?). Then folks can test easily, I don't see how it'd break stuff for anyone not using the new profiles?

The exception is that I'm keen to rename / remove the arm profile ASAP as I think it causes more confusion than solves problems.

Then I would say that the next step is to add commands to nf-core/tools to automate the config generation. Then we can roll it out more widely, where modules have ARM containers. In parallel we should check in on how that's going.

Basically I'd like to build out as much of this as possible in a non-destructive way, then change the profile names when we're ready.

Thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ewels ewels ewels left review comments

@mashehu mashehu mashehu left review comments

@FriederikeHanssen FriederikeHanssen Awaiting requested review from FriederikeHanssen

At least 1 approving review is required to merge this pull request.

Assignees

@edmundmiller edmundmiller

Labels
None yet
Projects
arm64 software builds
Status: Waiting to be merged
nf-core infrastructure projects
Status: In Progress
Milestone
3.19.0
Development

Successfully merging this pull request may close these issues.
4 participants
@edmundmiller @nf-core-bot @ewels @mashehu