[NX Release]: Full CI/CD automation with independent fixed versioned nx workspace libraries and apps using typescript project references

[electrocnic]

I dug a little bit around with available tools and practices, and for now it looks like the following usecase scenario is not fully supported (neither by nx release, nor npm, nor changesets, nor other tools or package managers?) Please correct me if I am wrong, I am happy to receive input here, this should be a bi-directional "blog" if you want.

I will work on something to make the workflow in the title possible and will share my progress and code here.

Problem Description

Initial Situation:

• NX Monorepo (Vite, Typescript, React, npm)
• Using Typescript Project References (https://nx.dev/recipes/tips-n-tricks/switch-to-workspaces-project-references)
• Using package.json in every lib (individual versioning)
• The combination of the last two leads to the following issue:
  • On the local machine, npm will resolve dependencies of a lib by local path references instead of using their fixed versions, which helps during development, as the IDE will always use the locally checked out (latest) code sources.
  • However, this makes it impossible to build lib A which depends on lib [email protected] (for example) to really pull and include version 0.1.2 from lib B, because the local source code might have progressed beyond 0.1.2 (lets say local source is 0.3.7). Building lib A would implicitly include lib [email protected] as dependency even if package.json from lib A contains B: 0.1.2 as dependency.
  • Deactivating local project references would allow us to use the correct prebuilt artifact with the referenced fixed version as dependency, but would also lead to a worse DX. Not to mention that changes in the source code of lib B would no longer lead to realtime compile errors or other highlighting because lib A ultimately is not linked to the local source of lib B but to the fixed downloaded artifact in node_modules, which does not change if we change the source of the same library.
  • For a real release with git tags and so on, I want to have reproducible builds, i.e. I want to be able to checkout the git tag, run nx build, and it should automatically download the correct dependency versions and not use the latest local source code. (Of course the automatic checkout would mean I have to script something, add npm i or so to the target).

In more Code Snippet examples from package-lock.json, and workspace tree, it looks like this:

monorepo/
├── package.json
├── package-lock.json
├── nx.json
├── libs/
│   ├── libA/
│   │   └── package.json
│   └── libB/
│       └── package.json
└── apps/
    ├── appA/
    │   └── package.json
    └── appB/
        └── package.json

libA package.json (simplified):

{
  "name": "@myorg/lib-a",
  "version": "0.2.0",
  "main": "src/index.ts",
  "type": "module",
  "dependencies": {
    "@myorg/lib-b": "0.1.2"
  }
}

libB package.json (simplified):

{
  "name": "@myorg/lib-b",
  "version": "0.3.7",
  "main": "src/index.ts",
  "type": "module",
  "dependencies": {}
}

{
  "name": "@myorg/app-a",
  "version": "1.0.0",
  "private": true,
  "main": "src/main.tsx",
  "type": "module",
  "dependencies": {
    "@myorg/lib-a": "0.1.5"
  }
}

{
  "name": "@myorg/app-b",
  "version": "1.1.3",
  "private": true,
  "main": "src/main.tsx",
  "type": "module",
  "dependencies": {
    "@myorg/lib-a": "0.2.0"
  }
}

Available released artifacts on the self hosted artifactory:

...
@myorg/[email protected]
@myorg/[email protected]
...
@myorg/[email protected]
@myorg/[email protected]
...
@myorg/[email protected]
@myorg/[email protected]

The above package.json examples are using the fixed versions that they depend on, not the latest versions that are available, so be not confused.

Locally they would still reference the latest source code directly in the workspace root package-lock.json:

...
"packages": {
  "node_modules/@myorg/lib-a": {
      "resolved": "libs/libA",
      "link": true
  },
  "node_modules/@myorg/lib-b": {
      "resolved": "libs/libB",
      "link": true
  },
  "node_modules/@myorg/app-a": {
      "resolved": "apps/appA",
      "link": true
  },
  "node_modules/@myorg/app-b": {
      "resolved": "apps/appB",
      "link": true
  },
}

To achieve the fixed version reproducible build, I would need to (and will work on a tool which will also use the nx release api) "switch" this npm mode from "link" to a url pattern.

For example, external npm dependencies are resolved like this:

"node_modules/react": {
      "version": "18.3.1",
      "resolved": "https://pkgs.dev.azure.com/our_own_azure_npm_registry/npm/registry/react/-/react-18.3.1.tgz",
      "integrity": "sha1-SauJIAnFOTNiW9FrJTP8dUyrKJE=",
      "license": "MIT",
      "dependencies": {
        "loose-envify": "^1.1.0"
      },
      "engines": {
        "node": ">=0.10.0"
      }
    },

... and so would my own published nx libs look like too.

Another Problem Detail

Using Vite with NX with typescript leads to the following import structure in the sources:

import { a } from '@myorg/lib-a'
import { b } from '@myorg/lib-b'

and whenever a dependency in a package.json to a local lib contains a package name that can be found in the local libraries' package.jsons (see above, I have libA and libB in my workspace filetree, and both contain a package.json with the identifier @myorg in it), running npm i will resolve all packages that match this @myorg name by local file references.

Therefore, it is not possible to simply add @myorg to the .npmrc or to delete the local file links from the package-lock.json, as npm i would simply generate this again as it would find all the libs in the folders again.

Renaming all the imports in all sources would be too error prone and cumbersome, even if automated. Also an absolute anti-pattern for ci-cd to change sources on-the-fly. It is already bad enough that I will have to script some on-the-fly changes during ci-cd release to switch to the artifact npm sources instead of local package links.

Solution Approach (Theory)

NPM Aliases

To leave the import statements in the source files untouched and still be able to switch the mode from file link to artifact download, I will likely publish my libaries to a slightly different npm package name: Instead of @myorg they will be named @myorg-published. Therefore, @myorg packages will always be local latest live source code libraries, and @myorg-published packages will always be prebuilt, bundled npm artifacts that are published to the registry.

An npm alias looks like this:

"dependencies": {
  "@myorg-published/lib-b": "0.1.2",
  "@myorg/lib-b": "npm:@myorg-published/[email protected]",
}

so @myorg/lib-b points to the published package with a specific version.
Running npm i now would no longer add file links, but instead really pull @myorg-published/[email protected] from the registry, yet the import statement still just references @myorg/lib-b.

Whenever a library should locally link a fixed artifact, this approach can locally work too, not just for the release pipeline. Then we can have some libs, that reference the latest sources in the workspace, and some that reference downloaded artifacts in node_modules.

Only thing is for now, nx release does not yet bump versions in npm aliases to my knowledge (not sure if it does in the meantime), would be a nice-to have, for now I must script this synchronization on my own.

Also, in order to switch back to the local reference, one has to remove the alias and revert the package reference from @myorg-published to @myorg.

A custom made nx tool will handle the switching before, during and after the release progress, and will incorporate the nx release api for the actual version bumps and changelog creation and git tagging and so on.

Automated Versioning

I want to have a convention that every merge to or commit on my dev-integration branch triggers a snapshot (prerelease) version, with no manual user intervention, and a commit or merge on the main branch triggers a normal release. I will implement a pipeline setup that will run nx release on the pipe, meaning it will commit and push on the pipe and trigger another pipeline. The second pipeline will then care about the actual build with the previously pushed library artifacts and release the app with the new version.

Practical implementation

I will continuously report my progress here. I would be interested in your feedback to this whole idea and approaches and if somebody already has something easier, more robust, or with better practices?