Purge unsupported adaptors in metadata cli with version-aware caching #953
Conversation
- Auto-remove adaptors that don't support metadata to prevent memory leaks - Cache unsupported adaptors per major.minor version to avoid unnecessary downloads - Add --keep-unsupported flag to opt out of automatic purging - Comprehensive test coverage for version comparison logic Addresses #952 and OpenFn/lightning#3351
packages/cli/src/metadata/cache.ts
Outdated
|
|
||
| const getPath = (repoDir: string, key: string) => `${repoDir}/meta/${key}.json`; | ||
| const CACHE_DIR = '.cli-cache'; |
There was a problem hiding this comment.
Phew, glad I checked this. This is incorrect.
.cli-cache is created by the CLI when running a workflow. It's created in the same folder as workflow.json, and it saves the outputs of each step in the workflow. Think of it like a local dataclip cache. It makes it easy to re-run a workflow from step 18 without having to manually build the right state object.
The metadata cache lives in a global CLI metadata folder called The Repo. This is where adaptors autoinstalled to.
Because the repo existed before the local dataclip cache, when I wanted a cache for the metadata service I added it to this repo. And to be fair I still think it's the correct behaviour - the metadata cache is global to a credential, if that makes sense.
So I'm pretty much going to revert this little change (which may also fix the broken tests)
in unit tests sometimes autoinstalled adaptors don't have a dependencies object. I don't think this affects anything in prod
Short Description
When Lightning requests metadata via the CLI, adaptors that don't support metadata are downloaded but never removed, wasting space and causing issues in container environments.
Fixes #952
Re: OpenFn/lightning#3351
Implementation Details
--keep-unsupportedflag allows opting out of automatic removalnpm uninstallinstead of manual file deletionQA Notes
Still needs to be hand tested!
AI Usage
Please disclose how you've used AI in this work (it's cool, we just want to know!):
You can read more details in our Responsible AI Policy
Release branch checklist
Delete this section if this is not a release PR.
If this IS a release branch:
pnpm changeset versionfrom root to bump versionspnpm installpnpm changeset tagto generate tagsgit push --tagsTags may need updating if commits come in after the tags are first generated.