[cye/subpack-ci] Add sub-package build, test, and publish to OSS. (WORK IN PROGRESS - PENDING MORE SUB-PACKAGE COVERAGE)

[cspades]

Description

• Prototype sub-package build, testing, and PyPI publication to BioNeMo OSS.
• GitHub Actions Workflow: https://github.com/NVIDIA/bionemo-framework/actions/workflows/bionemo-subpackage-ci.yml

TODO

• Get GPU support for GitHub Actions Runners. Then onboard GPU-dependent packages to singleton pytest-ing.
  • Depends on switch to GHA runners #734 @pstjohn
• Support installation of Apex, transformer_engine, mamba-ssm, and pyg, which requires pre-installation of torch and CUDA.
  • Probably can put this into a future PR. We're also investigating "better" installation methods, like for TE here: https://nvidia.slack.com/archives/C03V462SAMS/p1738598945887399 @pstjohn
• Post-Merge: Test the merged workflow_dispatch.
  • Do not merge until we secure GitHub Actions for BioNeMo FW only: https://docs.github.com/en/organizations/managing-organization-settings/disabling-or-limiting-github-actions-for-your-organization
  • Need to specify permissions for who can run the post-merge publication workflow to PyPI. Likely just specific individuals or any NVIDIAN because it is based on main branch, which presumably has "working" code.
• Cut or update a release tag for each sub-package when the sub-package is published to PyPI.
  • Could not get this to work, kept getting error code 422, i.e. invalid or endpoint spam.

Details

• This workflow has two paths: pull_request and workflow_dispatch.
  • For PR's, we always install, test, build, and publish to Test PyPI.
    • Requiring manual approvals in the testpypi GitHub Environment is typically unnecessary as it’s designed to run on each commit to the main branch and is often used to indicate a healthy release publishing pipeline. We simply want to check that we can either push to test PyPI, and if the version exists, that the duplicate version is detected properly on PyPI and publication is skipped.
    • Current state of the workflow should not block any PR's due to testing or publication. Tests incompatible with the workflow (i.e. no GPUs, cannot be installed, cannot be built, etc.) are automatically ✅ .
  • For workflow dispatches, we install, test, and build the comma-separated sub-package names, e.g. bionemo-core,bionemo-evo2,..., and optionally publish the packages to PyPI when we are ready to cut a release tag for the sub-package.
• Versioning
  • When publishing, we skip-existing versions of packages, so we don't ever overwrite past versions of a package. To update the version, we now require developers to change the VERSION file in the sub-package, which will no longer be linked to the BioNeMo Framework VERSION that only represents "working combinations" of ALL sub-packages in our release containers on NGC. Dysfunctional versions of published packages should be deprecated by the package maintainer(s) on PyPI.
  • Versions have been manually set to new "starting points" that are independent of BioNeMo FW, and most packages besides bionemo-fw, bionemo-testing, bionemo-example_model, and bionemo-llm will follow their own versioning history.
• Approximately 50% of this PR was written or informed by Cursor/Claude-3.5-Sonnet, the jq parsing, git diff parsing, the strategy matrix and output passing, and the branch definitions like ${{ github.event_name == 'pull_request' && 'testpypi' || 'pypi' }}.
• GHA PyPI Reference: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

Sub-Package Solo Build Changelog

⚠️ = Skipped singleton testing, and push to PyPI anyways. When testing is supported, PIC intervention to make the singleton install work will be necessary! Published versions are NOT EXPECTED TO FUNCTION beyond the BioNeMo Framework Container™️ !
☑️ = Skipped publication, i.e. PyPI publish isn't necessary or the package cannot be distributed as a PyPI package.
❌ = Neither singleton testing nor publication are supported or necessary by the workflow. Skipped gracefully to prevent blocking PR merge criteria and multi-package workflows.

• bionemo-core ✅
  • Fix some dependencies not explicitly installed, and updated import paths.
  • Version: Bump to 2.4.0 for minor fixes and to mirror our BioNeMo FW release version, as it contains important resources required by the other sub-packages.
• bionemo-moco ⚠️ @nvdreidenbach
  • Requires GPU for unit tests.
  • Version: Same as latest published. (0.0.2)
• bionemo-scdl ✅ @polinabinder1
  • Version: 0.0.7
  • Depends on bionemo-core>=2.2.1, which was published ad hoc to fix the GitHub Workflow by @polinabinder1.
• bionemo-noodles ❌ @skothenhill-nv
  • Requires uv install maturin for pytest and publication to PyPI.
  • Version: 0.0.1
• bionemo-geometric ❌ @sveccham
  • Requires pre-installation of PyTorch: https://pytorch-geometric.readthedocs.io/en/latest/install/installation.html
  • Publication not required according to @sveccham.
• bionemo-webdatamodule ⚠️ @DejunL
  • Requires GPU for unit tests, likely all other tests pass.
  • Version: 1.0.0
• bionemo-testing ☑️
  • Should not be published to PyPI as per pyproject.toml description.
  • Testing is still run on it.
• bionemo-llm ⚠️ @jstjohn
  • Requires pre-installation of PyTorch, TE, Apex, etc.
    • Which bring in further dependencies like: https://github.com/state-spaces/mamba?tab=readme-ov-file#installation
  • Version: 2.4.0
• bionemo-size-aware-batching ⚠️ @DejunL
  • Requires GPU for unit tests, likely all other tests pass.
  • Version: 1.0.0
• bionemo-evo2, bionemo-esm2, bionemo-geneformer ⚠️ @jstjohn @farhadrgh @sichu2023 @pstjohn @skothenhill-nv
  • Version: 2.4.0
  • Cannot test without GPU's, publishing the builds for container installation.
• bionemo-fw ⚠️
  • Version: 2.4.0
  • Cannot be tested/installed because it depends on everything else. Testing is skipped until we have everything else published.
• bionemo-example_model ❌
  • Does not need to be published, and cannot be tested without GPU.

Type of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor
• Documentation update
• Other (please describe):

Usage

BioNeMo PyPI Sub-Package Onboarding

To onboard your sub-package to the bionemo-subpackage-ci.yml workflow:

• Log in to PyPI and Test PyPI.
• Go to Account Settings > Publishing, and either...
  • If your package already exists on PyPI: Manage [Package Name] > Add a New Publisher.
  • If your package does not yet exist on PyPI: Add a New Pending Publisher.
• Under the GitHub tab of the publisher management page, input the following information:
  • Owner: NVIDIA
  • Repository Name: bionemo-framework
  • Workflow: bionemo-subpackage-ci.yml
  • Environment Name:
    • pypi for PyPI
    • testpypi for Test PyPI
• Run the workflow!
  • Create or update any PR with git diff changes to your bionemo-framework/sub-packages.
    • Publishes to Test PyPI.
  • Dispatch the bionemo-subpackage-ci.yml workflow from GitHub Actions.
    • Publishes to PyPI.
    • Required: Input a comma-separated list of sub-packages you want to test and/or publish into subpackages.
    • Optional: Set publish to true if you want to publish to PyPI. (Default: false)

BioNeMo PyPI Sub-Package Off-Boarding

In case the workflow is not compatible with your sub-package, such as your sub-package tests needing GPU's or more complicated installation strategies, you can completely deactivate this workflow for your sub-package by simply updating two ENV variables in the workflow YAML: bionemo-framework/.github/workflows/bionemo-subpackage-ci.yml.

env:
  # TODO(@cspades): Skip pytest for packages that require GPU's or special installation recipes until resolved.
  SUBPACKAGE_SKIP_TEST: '["bionemo-moco","bionemo-noodles","bionemo-geometric","bionemo-webdatamodule","bionemo-size-aware-batching","bionemo-testing","bionemo-llm","bionemo-evo2","bionemo-esm2","bionemo-geneformer","bionemo-example_model","bionemo-fw"]'
  # TODO(@cspades): Skip publishing for packages that require special installation recipes or should not be published to PyPI.
  SUBPACKAGE_SKIP_PUBLISH: '["bionemo-noodles","bionemo-geometric","bionemo-testing","bionemo-example_model"]'

Testing

• Tested the PR pipeline on supported sub-packages in this PR, which now exists on Test PyPI. When merged, the workflow dispatch can then publish to PyPI.

# When you attempt to run the publish workflow when a version already exists on PyPI and `skip-existing: true`.
WARNING  Skipping bionemo_core-2.2.1-py3-none-any.whl because it appears to     
         already exist

[trvachov]

Discussed via huddle -- a great first step forward.

[cspades]

@cspades To confirm, when deployed this will push e.g. bionemo-llm ?

PR changes push to test PyPI, and manual workflow dispatches are required to push to the real PyPI.

I'll be testing with @nvdreidenbach 's bionemo-moco package since he has everything in PyPI setup for this workflow. Like in the PR tests, it won't publish anything because 0.0.2 already exists.

Edit: Or bionemo-core.

[pstjohn]

overall LGTM with comments discussed via slack; let's make this manual for now (not based on PR) until we're more comfortable with pypi distribution