Walkdir caching first pass

[RomneyDa]

Description

Cache listDir results and ignore file rectified ignores for directories

• for 30 seconds OR
• till invalidated

Any file changes, creations, deletions, or index requests will invalidate the cache, so the only effect this has for now is that parallel walk dirs will be much more efficient. It will not speedup e.g. walkdir for a single instance of codebase reindexing.

Ways to improve:

• More precise invalidation logic and/or longer cache time
• this approach does not make the walk dir significantly faster, it just makes way more processor-efficient. Parallel walkDir calls still wait for the promises of previous ones. This might be preferable to a shorter but more severe spike. I tried some approaches to saving cache hits for later and snagging the next

Case study: Startup in react repo

Before:

After:

CPU Load:

[netlify]

✅ Deploy Preview for continuedev canceled.

Name	Link
🔨 Latest commit	14db334
🔍 Latest deploy log	https://app.netlify.com/sites/continuedev/deploys/67d4756d156c6f0008a1b664

[RomneyDa]

This is another improvement - if one ignore comes out as true, skip the rest.