Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: improve globSync performance #57725

Merged
merged 1 commit into from
Apr 7, 2025
Merged

fs: improve globSync performance #57725

merged 1 commit into from
Apr 7, 2025

Conversation

Trott
Copy link
Member

@Trott Trott commented Apr 2, 2025

Benchmark results:

                                                                                    confidence improvement accuracy (*)   (**)  (***)
fs/bench-glob.js recursive='false' mode='async' pattern='*.js' dir='lib' n=1000                    -0.28 %       ±2.63% ±3.50% ±4.56%
fs/bench-glob.js recursive='false' mode='async' pattern='**/*' dir='lib' n=1000                     0.15 %       ±2.85% ±3.81% ±4.98%
fs/bench-glob.js recursive='false' mode='async' pattern='**/**.js' dir='lib' n=1000                -1.03 %       ±2.85% ±3.79% ±4.94%
fs/bench-glob.js recursive='false' mode='sync' pattern='*.js' dir='lib' n=1000                     -0.49 %       ±1.39% ±1.86% ±2.42%
fs/bench-glob.js recursive='false' mode='sync' pattern='**/*' dir='lib' n=1000               *      0.82 %       ±0.68% ±0.91% ±1.19%
fs/bench-glob.js recursive='false' mode='sync' pattern='**/**.js' dir='lib' n=1000         ***      1.38 %       ±0.73% ±0.98% ±1.29%
fs/bench-glob.js recursive='true' mode='async' pattern='*.js' dir='lib' n=1000                      0.31 %       ±2.33% ±3.09% ±4.03%
fs/bench-glob.js recursive='true' mode='async' pattern='**/*' dir='lib' n=1000                     -0.90 %       ±3.45% ±4.59% ±5.98%
fs/bench-glob.js recursive='true' mode='async' pattern='**/**.js' dir='lib' n=1000                  0.26 %       ±2.46% ±3.28% ±4.26%
fs/bench-glob.js recursive='true' mode='sync' pattern='*.js' dir='lib' n=1000                       0.01 %       ±0.92% ±1.23% ±1.60%
fs/bench-glob.js recursive='true' mode='sync' pattern='**/*' dir='lib' n=1000                       0.57 %       ±0.62% ±0.83% ±1.09%
fs/bench-glob.js recursive='true' mode='sync' pattern='**/**.js' dir='lib' n=1000           **      1.89 %       ±1.35% ±1.82% ±2.41%

Be aware that when doing many comparisons the risk of a false-positive result increases.
In this case, there are 12 comparisons, you can thus expect the following amount of false-positive results:
  0.60 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.12 false positives, when considering a   1% risk acceptance (**, ***),
  0.01 false positives, when considering a 0.1% risk acceptance (***)

@nodejs-github-bot nodejs-github-bot added fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels Apr 2, 2025
@Trott Trott added request-ci Add this label to start a Jenkins CI on a PR. author ready PRs that have at least one approval, no pending requests for changes, and a CI started. labels Apr 2, 2025
Copy link

codecov bot commented Apr 2, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.22%. Comparing base (1c2d98d) to head (7367a0b).
Report is 72 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #57725      +/-   ##
==========================================
- Coverage   90.22%   90.22%   -0.01%     
==========================================
  Files         630      630              
  Lines      185073   185070       -3     
  Branches    36222    36223       +1     
==========================================
- Hits       166990   166982       -8     
+ Misses      11044    11041       -3     
- Partials     7039     7047       +8     
Files with missing lines Coverage Δ
lib/internal/fs/glob.js 91.97% <100.00%> (-0.04%) ⬇️

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Apr 2, 2025
@nodejs-github-bot
Copy link
Collaborator

// if it matches be more performant in the typical use case? #isExcluded()
// is often ()=>false which is about as optimizable as a function gets.
if (this.#cache.statSync(fullpath).isDirectory() && this.#isExcluded(`${fullpath}/`)) {
if (this.#isExcluded(`${fullpath}/`) && this.#cache.statSync(fullpath).isDirectory()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the fastest way to would be to use this function: https://github.com/nodejs/node/blob/main/src/node_file.cc#L1053

Particularly: internalBinding('fs').internalModuleStat(path) which avoids creating unnecessary Dir/Dirent objects.

@nodejs-github-bot
Copy link
Collaborator

@aduh95 aduh95 merged commit b2ab1a2 into nodejs:main Apr 7, 2025
68 checks passed
@aduh95
Copy link
Contributor

aduh95 commented Apr 7, 2025

Landed in b2ab1a2

@Trott
Copy link
Member Author

Trott commented Apr 7, 2025

I'll take a look at @anonrig's suggestion for a subsequent PR (unless someone beats me to it!).

@Trott Trott deleted the glob-perf branch April 7, 2025 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants