Target aware "affected" functionality

[skynetigor]

Description

NX is an incredible tool to manage monorepo. It quite well solves problems with long CI runs. But I think it can work even better.
I'll make a few examples based on unit-tests.

Motivation

Current behavior:
Let’s imagine we have a book-store application and a book-core lib that book-store depends on.
Whenever I change only .spec.ts file in book-core, nx print-affected --target=build compares HEAD (a spec changed) with the BASE and returns book-store project even though book-store does not have files depending on any specs from book-core. Consequently, it will trigger the build of book-store even though book-core was not changed (what was changed is just its tests)

Suggested behavior:
If I change just a spec file for book-core this change should not affect book-store since these changes are just unit-tests (not changes to the lib implementation) and hence should not trigger build of book-store.

Suggested Implementation

nx print-affected --target=build should determine whether “build” related files changed in dependency, and:
if there are such files - treat dependent project with “build” target as affected
otherwise, treat the project as not affected

To let NX be aware of what exactly files relate to a specific target we could add include/exclude properties glob patterns.
For example, for project book-core in “build” target there could be added “targetUnrelatedFiles” [“**/*.spec.ts”] exclusion and when NX will build a dependency graph for book-store it will skip book-core project as there hasn’t been any “build” related files in book-core.

Targets that might need that functionality:

• Unit-tests - when changes are made only in *.spec.ts files
• Storybook - when changes are made only in *.stories.ts files

Alternate Implementations

In the current implementation, It could be achieved by adding a separate project per target.
For example, for book-core you could add book-core-unit-tests lib and make dependency to book-core lib in order to be able to test its modules. This way it will work as expected. The main downsides of it is the growing number of projects and you can’t write UTs for encapsulated services and components.

[rlmestre]

Since 14.4, you can specify inputs on targets globally or per-project. This allows you to control fileset inclusion/exclusion for the task. Does that cover your use-cases?

https://nx.dev/more-concepts/customizing-inputs

[ihor-panasiuk95]

As per the documentation, this is more about NX Cache and there is no any mentions about whether it participates in "affected" computation.
So if it does, this more a documentation issue rather than feature request.

[ihor-panasiuk95]

But if it doesn't, I guess reusing of this inputs for "affected" computation along with caching looks good.
Thanks for the reply!

[evanliomain]

I made some experiments with inputs, and I confirm it only control the Nx Cache detection.

I am also interested in this feature to launch only libs with significant changes.

Today, Nx executor relies on affected libs computation and cache system. Only the cache can be fine tune with inputs. It would be great is affected libs will be too.

[chrisui]

What does the --target X option do within nx print-affected command?

[skynetigor]

What does the --target X option do within nx print-affected command?

https://nx.dev/packages/nx/documents/print-affected#targets
it prints affected apps by a specific target (build, test, storybook, e2e, etc)

[tomdglenn91]

Posting for visibility as googling if this was possible brought me here. Nx's greatest strength is in saving time in CI, and this would be a huge huge boost. If I make a PR that only updates tests, and my CI server not running build, would be incredible.

[ianldgs]

Posting for visibility as googling if this was possible brought me here. Nx's greatest strength is in saving time in CI, and this would be a huge huge boost. If I make a PR that only updates tests, and my CI server not running build, would be incredible.

In theory that already happens because of build cache. Test files are typically not inputs of the build target. Module is affected but the build runs instantly.

[tomdglenn91]

That's only the case if I have a cache to read from. We're currently not set up for nx cloud (early days of integrating nx) so i've got a cold app on all fronts.

[mirobo]

We have 170+ projects and each of the project also has e2e tests as well as Angular code that is tested with these e2e tests. Since "nx affected" detects only if a project (in its whole) is affected, we have the problem that changing e2e tests results in a rebuild of the app.
We could create a separate project for each library to account for that. But it will not resolve the underlying problem, since we also have MD-files in the project that need to be linted, if they were changed, unit tests that need to be run, if they were changed (without a rebuild of the app). As previously mentioned we could boost CI build times massively if NX affected is "target-aware".

[FrozenPandaz]

I do agree that affected could be smarter. However, I don't know when we will get to this. There's a lot of design which will have to go into this.

I know it's not the same but tasks being cached is a huge mitigation here. Though there will be affected tasks, if they haven't changed, they should be cached. Hence we've prioritized other things first. Again, not the same.. but very much yields fast pipelines.

[skynetigor]

The problem with relying only on cache is that you will anyway have to run CI job for unaffected project, you will need to setup dependencies, do some other prerequisites (sometimes they might take long and hence expensive, depending on the project) and only on build step NX will understand that nothing affected relatively to build and it can restore build for the project from cache.

Maybe, if NX could rely on Cache during "affected" functionality (maybe it's the easiest way to achieve what described in this issue), like check in cache if cache exists for project (in cloud I think) then project is not "affected", otherwise, if cache does not exist, the project is affected.

In our project we have 12 MFE applications, cypress project per each app, and dozens of shared libraries with unit tests.
We managed to achieve the desired behavior but leveraging "hash" property that NX provides.
In CI workflow, we have one Job called "Compute affected projects", that computes what projects are affected and then distribute affected projects though matrix jobs.
Then, each CI job that builds/tests a project, saves output of build/test to GCP bucket with this "hash". In the next workflow run, "Compute affected projects" computes NX hash again and checks in GCP bucket if cache exists. If so, project is not affected. Otherwise - affected. Additionally, we have ability to restore all our project and glue up all our MFE build results into one artifact by just restoring all build caches relative to the current run. We use restore multiple times: during deployment and to run Cypress tests in separate jobs.

NOTE: we don't use "affected" at all. We always rely on Cache in GCP bucket.

Hope I described it clear and it would help to improve your build process.
I think it's worth writing a blog about this process.

Nevertheless, NX is doing great job, I really love this concept and it brings monorepo on a new level. Don't like Micro-repo since I've got familiar with NX. 😌

[akwodkiewicz]

In my use-case, I am also distributing the matrix jobs on GHA runners using nx show projects --affected --with-target=build output. Each item gets assigned a separate job and the whole thing is parallelized. Basically what @skynetigor wrote above. But we don't have the cache part (yet).

I planned on making this even "smarter", by excluding all .md files from build target's inputs, so that when I only modify the README.md in a PR, the CI will not build anything. But in the current Nx's behaviour it seems that this inputs exclusion does not do much.

The only way to achieve the "smart" part is to start dumping the cache and restore it over and over again, but these are additional steps, that have to be maintained, require storage, networking and, most importantly, they take additional time!

Obviously, not running a task at all is going to be much quicker than restoring cache from some storage bucket, unpacking it and then deciding that the build can be reused from the cache.

Command

nx show projects --affected --with-target=build

Current behaviour

List projects that had any of their files changed AND they have a build target defined

Expected behaviour

List projects for which build target's inputs were modified.

[akwodkiewicz]

List projects that had any of their files changed AND they have a build target defined

BTW, when is a file considered to be part of the project? I could only find this in the docs:

When you run nx affected -t test, Nx will:

• Use Git to determine the files you changed in your PR.
• Use the project graph to determine which projects the files belong to.
• Determine which projects depend on the projects you modified.

But there's nothing about "belongingness" of a file to a particular project. I assume that the directory structure is what matters here, but there is no confirmation of it anywhere. Project graph docs (linked in the quote) are also more about "task graph" not the files themselves.

If there was a way to exclude some files from the project completely, I could also try doing that, although those README.md files could never trigger any other target. Maybe I can use .nxignore, as a global approach.

Still, I think that nx show --affected --with-target (the original request is about print-affected, but it got deprecated) should not only consider cache usage, but if the task should be actually run.

[akwodkiewicz]

assume that the directory structure is what matters here, but there is no confirmation of it anywhere

Yeah, I found this util in the source code, it iteratively strips parts of the file path in search for the project root directory:

nx/packages/nx/src/project-graph/utils/find-project-for-path.ts

Lines 43 to 64 in 355d5c5

	export function findProjectForPath(
	filePath: string,
	projectRootMap: ProjectRootMappings
	): string | null {
	/**
	* Project Mappings are in UNIX-style file paths
	* Windows may pass Win-style file paths
	* Ensure filePath is in UNIX-style
	*/
	let currentPath = normalizePath(filePath);
	for (
	;
	currentPath != dirname(currentPath);
	currentPath = dirname(currentPath)
	) {
	const p = projectRootMap.get(currentPath);
	if (p) {
	return p;
	}
	}
	return projectRootMap.get(currentPath);
	}

[github-actions]

This issue has been automatically marked as stale because it hasn't had any activity for 6 months.
Many things may have changed within this time. The issue may have already been fixed or it may not be relevant anymore.
If at this point, this is still an issue, please respond with updated information.
It will be closed in 21 days if no further activity occurs.
Thanks for being a part of the Nx community! 🙏

[github-actions]

This issue has been closed for more than 30 days. If this issue is still occuring, please open a new issue with more recent context.