Skip to content

Commit 7df690a

Browse files
Patrick-ErichsensestinjpriyashpatildevbyjonahYohannZe
authored
Dev (#1784)
* Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <[email protected]> Co-authored-by: Priyash <[email protected]> Co-authored-by: Jonah Wagner <[email protected]> Co-authored-by: YohannZe <[email protected]> Co-authored-by: Dan Dascalescu <[email protected]> * temporary patch for type errs * Dev (#1689) * Fix an issue where CMD+K does not clear the terminal when the terminal has focus (#1671) On MacOS, ⌘+K is bound, by default, to Terminal:Clear. Without this change ⌘+K does not clear the terminal but instead iniates a chord sequence and waits for the next stroke of the chord. Co-authored-by: Rob Leidle <[email protected]> * Change treeSitter to cache the Language objects it loads from wasm (#1672) Without this change, for a repository with 600 typescript files, the indexer would fail to finish correctly and there would be many of the following errors in the webview console log: 'Unable to load language for file ${path} RuntimeError: table index is out of bounds' The following bash will create a repo that reproduces the problem: current_path="." for ((i=1; i<=20; i++)); do new_folder="folder-$i" mkdir -p "$current_path/$new_folder" current_path="$current_path/$new_folder" for ((a=1; a<=30; a++)); do head -c 10000 /dev/urandom | base64 > "$current_path/file-$a.ts" done done Co-authored-by: Rob Leidle <[email protected]> * acknowledge sourcemap flag in esbuild.js * don't run jetbrains-release.yaml on vscode releases * further testing for walkDir * chore: add telemetry to commands (#1673) * test: Add basic unit test to baseLLM (#1668) * update version * test: Add basic unit test to baseLLM --------- Co-authored-by: Nate Sesti <[email protected]> Co-authored-by: inimaz <[email protected]> * feat: add Quick Actions CodeLens feature (#1674) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: add quick actions * Update index.d.ts * quick actions mvp * update docs * subscribe to vscode change settings * Update commands.ts * cleanup * Update quick-actions.md * Update VerticalPerLineCodeLensProvider.ts * resolve feedback --------- Co-authored-by: Nate Sesti <[email protected]> Co-authored-by: Nate Sesti <[email protected]> Co-authored-by: Jonah Wagner <[email protected]> * chore: add `isCommandEvent` to command telemetry (#1675) * chore: add `isCommandEvent` to command telemetry * Update commands.ts * Nate/better retrieval (#1677) * deduplicatearray tests * break out separate retrieval pipelines * IConfigHandler * tests for codebase indexer * better .continueignore for continue * indexing fixes * ignore .gitignore and .continueignore when indexing * retrieval pipeline improvements * fix formatting err in out .continueignore * add necessary filter to lance_db_cache * update package.json version * skip unused tests * don't ignore .prompt files * update version * Update pull_request_template.md * don't use multi-media format when there are multiple text items * add free trial experience (#1685) * fix: add code range for quick actions/fixes (#1687) * fix: add code range for quick actions/fixes * Update test.js * add pathSep message type * docs improvements * jetbrains fix * update package.json version --------- Co-authored-by: Rob Leidle <[email protected]> Co-authored-by: Rob Leidle <[email protected]> Co-authored-by: Patrick Erichsen <[email protected]> Co-authored-by: inimaz <[email protected]> Co-authored-by: inimaz <[email protected]> Co-authored-by: Jonah Wagner <[email protected]> Co-authored-by: Priyash <[email protected]> * update package.json version * skip extension tests * update package.json version * handlebars import * don't push package.json version change from CI * fix: scroll issues w/ code blocks (#1688) * fix: scroll issues w/ code blocks * chore: name offset * chore: remove docs start * chore: update headings on troubleshooting.md (#1696) * ignore .svn folder for indexing (#1699) * chore: clearer naming for `useSuffix` (#1702) * chore: clearer naming for `useSuffix` * schema updates * fix: cmd+shft+l closes sidebar if focused (#1638) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * add voyage rerank-1 * import Handlebars * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * small UI tweaks * media query * feat: add best experience onboarding * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * chore: add telemetry for full screen toggle (#1618) * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * manually implement stop tokens for hf inference api * chore: onboarding metrics (#1626) * fix: pageview tracking * feat: add onboarding telemetry * create single `onboardingStatus` type * improved var naming * remove console logs * fix windows performance issue * rename vscodeExtension.ts * migration of onboarding variables * "stash" instead of "delete" in indexing progress * fix preview.yaml * also fix main.yaml * Update troubleshooting.md (#1637) * feat: close panel if main input is focused * add skip param * Update TipTapEditor.tsx * merge dev * Update commands.ts --------- Co-authored-by: Nate Sesti <[email protected]> Co-authored-by: Nate Sesti <[email protected]> Co-authored-by: Jonah Wagner <[email protected]> * bugfix: quick actions undefined array (#1704) * feat: add rich quick pick for quick edit (#1706) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * Create deepseek.md (#1708) * style: add max height to model selector (#1712) * update styling * remove testing code * remove comments * feat: add tutorial card (#1716) * feat: create tutorial card * Update TutorialCard.tsx * feat: add file search to quick edit (#1714) * feat: multistep quick pick * fix: multi step for context provider * Update test.js * add file search * complete file search * remove unused import * add comments * Update utils.ts * update naming * chore: move quick pick instantiation (#1723) * Break out into NPM modules (#1726) * config.json types * export all schemas * break out request logic into packages * update openai adapters to export body types * update packaging of npm modules * update packages * continue proxy * mistral * Fix typo in documentation (#1722) In set-up-codestral.md "Trobleshooting" -> "Troubleshooting" * update package.json version * 🪄 model dropdown, control plane (#1692) * Nate/control plane client (#1691) * docs: add docs and schema for "OS" provider (#1536) * ignore .env * fix(gui): ctx rendering w/ renderInlineAs: "" (#1541) * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * fix codeqwen autocomplete leading space * add voyage rerank-1 * feat: `--noEmit` for tsc checks in CI (#1559) * docs: update CustomContextProvider docs (#1557) * add stop tokens to qwen prompt * update docs to reflect 3.5 sonnet being best * docs: comment out unused providers (#1561) * import Handlebars * feat: toast notification for config updates (#1560) * feat: toast notification for config updates * feat: only trigger toast on config.json save * displayRawMarkdown option * feat: open pane on install (#1564) * feat: open pane on activation * comment out testing code * update to reflect 16 stop words limit for deepseek * feat: only trigger config update toast in vscode (#1571) * docs(prompt-files): fix typos + clarify (#1575) * doc: prompt file typo + clarifications * fix: add back correct docs * chore: add telemetry for pageviews (#1576) * feat: update onboarding w/ embeddings model (#1570) * chore(gui): remove unused pages * feat: add embeddings step * feat: update styles * feat: copy button updates * fix: correct pull command for embed model * fix: remove commented code * fix: remove commented code * feat: simplify copy btn props * chore: rename onboarding selection event * feat: add provider config * fix: undo msg name * remove dead code * fix: invalid mode check * fix: remove testing logic * docs(telemetry): add pageviews to tracking list (#1581) * Add reranker configuration options to codebase embedding docs (#1584) - Introduce reranker concept - List available reranker options - Provide configuration instructions - Update keywords to include "reranker" * chore: update pr template with screenshots (#1590) * Refactor ConfirmationDialog to use SecondaryButton for cancel action (#1586) * Added instructions for running docs server locally (#1578) - Added NPM script method - Added VS Code task method - Update contributing guidelines * Update branch policy (#1577) - Change PR target to `dev` branch - Update `CONTRIBUTING.md` instructions * Consolidate example configurations into the main configuration guide (#1579) - Moved examples to configuration.md - Deleted the separate examples.md file - Updated sidebar order and links - Improved readability and structure in configuration.md * fix: fullscreen gui retains context when hidden, fixed fullscreen focusing (#1582) * Update completionProvider.ts (warning tab-autocomplete models) (#1566) * feat: enhanced IndexingProgressBar with blinking dot feature - Integrated BlinkingDot component - Added STATUS_COLORS for various states - Replaced CircleDiv with BlinkingDot in UI - Updated status messages and layout * small UI tweaks * feat(gui): enhance ModelCard, ModelProviderTag, and Toggle components (#1595) - add styling and adjustments to ModelCard - update ModelProviderTag font size - remove box shadow from Toggle component - tweak icon styles in ModelCard - improve alignment and spacing * media query * feat: add best experience onboarding * fix: file rename * stop movement on button hover by keeping same border thickness * fix mistake in setting cursor: pointer * fix when free trial option is shown * Support Node.js versions below 20 for streaming response handling (#1591) - Add fallback for Node < 20 - Implement toAsyncIterable for streaming - Use TextDecoder for manual decoding - Maintain existing streaming for Node 20+ * small fixes * feat: add free trial card to onboarding (#1600) * feat: add free trial card to onboarding * add import * fix hasPassedFTL * fix /edit cancellation from UI * feat: add `applyCodeBlock` experimental prop (#1601) * feat: add new model styling improvements (#1609) * feat: add new model styling improvements * better gap size * feat: update bug_report.yml (#1610) * chore: update bug_report.yml * typo fix * feat: add labels to "Add docs" dialog (#1612) * feat: add labels to "Add docs" dialog * remove autofocus * don't double load config * small fixes * speed up directory traversal, and use correct native path module * option not to show config update toast * merge air-gapped and recommended setup pages * chore: add telemetry for full screen toggle (#1618) * Fix headings in codebase-embeddings.md (#1617) * mention jetbrains * docs: update changie (#1619) * feat: updated changie config * hide toc and autogenerate * Update changelog.mdx * link to deeper explanation of embeddings models * ensure target="_blank" for all links in sidebar * fix gif links in intellij README.md * don't require rust in dependency installation * chore: fix padding on gh button (#1620) * chore: adjust button padding * Update tasks.json * escape colons in diff path * smoother lancedb indexing reporting * smooth progress updates for indexing * fix tsc err * rerank-lite-1 * remove doccs * basic tests for VS Code extension * improved testing of VS Code extension * docs: add docs and schema for "OS" provider (#1536) * ignore .env * 🚑 fix constant warnings when onboarding with Ollama * ✨ use and cache imports for autocomplete (#1456) * ✨ use and cache imports for autocomplete * fix tsc * team analytics * apply control plane settings * workos auth * ide protocol get session info * UI for auth * profile switching * small fixes * updates * refresh tokens * updates * fix tsc errs * model select in toolbar to make room for profile selector * prod client id * link to prod URL * internal beta option * profiles change listener --------- Co-authored-by: Patrick Erichsen <[email protected]> Co-authored-by: Priyash <[email protected]> Co-authored-by: Jonah Wagner <[email protected]> Co-authored-by: YohannZe <[email protected]> Co-authored-by: Dan Dascalescu <[email protected]> * temporary patch for type errs --------- Co-authored-by: Patrick Erichsen <[email protected]> Co-authored-by: Priyash <[email protected]> Co-authored-by: Jonah Wagner <[email protected]> Co-authored-by: YohannZe <[email protected]> Co-authored-by: Dan Dascalescu <[email protected]> * skip extension tests * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * fix: wrap editor buttons when out of space (#1727) * Add .continuerc file to config directory on activation to prevent it's indexing. (#1733) * Updated the content links that were deprecated in the document (#1709) Co-authored-by: Ihe Fan <[email protected]> * Nate/dev (#1739) * JetBrains status bar spinner for autocomplete * Remove unused images * partial accept jetbrains * fix double-rendering of jetbrains completions * fix offset of multi-line jetbrains completions * enable/disable actions * upgrade to version 0.0.54 * llm-info package * small profile switching improvements * instruct cmd+I not to leave placeholders * @ files in context-providers docs * onboarding fix * update jetbrains ci to build gui * feat: single default quick pick to edit (#1743) * feat: single default quick pick to edit * Update quick-actions.md * Update quick-actions.md * use defaultQuickAction * quick fixes * fix: handle line removal diff (#1744) * fix: handle line removal diff * remove comments * Configurable `maxChunkSize` for embedding providers (#1746) * Add `maxChunkSize` to `EmbeddingsProvider` * Add `maxChunkSize` to schema * add support for docs via config (#1594) * mistral mamba (#1748) * 🐍 mistral mamba * autoscale retrieval based on context length * autoscale retrieval based on context length * fix input box movement * update jetbrains build * test: add myers diff tests (#1754) * feat: enhance help center (#1755) * feat: enhance help center * Update help.tsx * feat: add azure provider config (#1764) * autocomplete reload fix * logout of control plane * Update QuickEditQuickPick.ts (#1772) * Update QuickEditQuickPick.ts * updates * Update QuickEditQuickPick.ts * pass refreshed access token to continue-proxy model * test: add dir read check for walkDir (#1773) * test: add dir read check for walkDir * Update ignore.ts * Update walkDir.test.ts * Update walkDir.test.ts * gpt-4o-mini * gpt-4o-mini * gpt4omini to ui selector * fix: layout alignment (#1779) * fix: delete old docs index on force re-index (#1778) * add "Codebase Force Re-Index" command (#1757) * Add `HuggingFaceTEIReranker` (#1711) * Add `HuggingFaceTEIReranker` * Add `huggingface-tei` to `config_schema.json` * update package.json version * update config schemas * skip test failing only due to type error * fix: convert `walkDir` to an async generator (#1783) * walkdir to async generator * add yields * handle ignore files * Update CodebaseIndexer.ts * Update walkDir.ts * chore: bump VS Code ext version --------- Co-authored-by: Nate Sesti <[email protected]> Co-authored-by: Priyash <[email protected]> Co-authored-by: Jonah Wagner <[email protected]> Co-authored-by: YohannZe <[email protected]> Co-authored-by: Dan Dascalescu <[email protected]> Co-authored-by: Nate Sesti <[email protected]> Co-authored-by: Rob Leidle <[email protected]> Co-authored-by: Rob Leidle <[email protected]> Co-authored-by: inimaz <[email protected]> Co-authored-by: inimaz <[email protected]> Co-authored-by: Anaël Bonnafous <[email protected]> Co-authored-by: Huy Tran <[email protected]> Co-authored-by: Gabriel Gordbegli <[email protected]> Co-authored-by: Raven-1027 <[email protected]> Co-authored-by: Ihe Fan <[email protected]> Co-authored-by: Lukas Kreussel <[email protected]>
1 parent 11a5f78 commit 7df690a

File tree

3 files changed

+116
-113
lines changed

3 files changed

+116
-113
lines changed

core/indexing/walkDir.ts

+114-111
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ class Walker extends EventEmitter {
6262

6363
emit(ev: string, data: any): boolean {
6464
let ret = false;
65+
6566
if (!(this.sawError && ev === "error")) {
6667
if (ev === "error") {
6768
this.sawError = true;
@@ -81,73 +82,67 @@ class Walker extends EventEmitter {
8182
return ret;
8283
}
8384

84-
start(): this {
85-
this.ide
86-
.listDir(this.path)
87-
.then((entries) => {
88-
this.onReaddir(entries);
89-
})
90-
.catch((err) => {
91-
this.emit("error", err);
92-
});
93-
return this;
85+
async *start() {
86+
try {
87+
const entries = await this.ide.listDir(this.path);
88+
89+
for await (const result of this.onReadDir(entries)) {
90+
yield result;
91+
}
92+
} catch (err) {
93+
this.emit("error", err);
94+
}
9495
}
9596

9697
isIgnoreFile(e: Entry): boolean {
9798
const p = e[0];
9899
return p !== "." && p !== ".." && this.ignoreFiles.indexOf(p) !== -1;
99100
}
100101

101-
onReaddir(entries: Entry[]): void {
102+
async *onReadDir(entries: Entry[]) {
102103
this.entries = entries;
104+
103105
if (entries.length === 0) {
104106
if (this.includeEmpty) {
105107
this.result.add(this.path.slice(this.root.length + 1));
106108
}
107109
this.emit("done", this.result);
110+
yield this.result;
108111
} else {
109112
const hasIg = this.entries.some((e) => this.isIgnoreFile(e));
110113

111114
if (hasIg) {
112-
this.addIgnoreFiles();
113-
} else {
114-
this.filterEntries();
115+
await this.addIgnoreFiles();
115116
}
117+
118+
yield* this.filterEntries();
116119
}
117120
}
118121

119-
addIgnoreFiles(): void {
122+
async addIgnoreFiles() {
120123
const newIg = this.entries!.filter((e) => this.isIgnoreFile(e));
121-
122-
let igCount = newIg.length;
123-
const then = () => {
124-
if (--igCount === 0) {
125-
this.filterEntries();
126-
}
127-
};
128-
129-
newIg.forEach((e) => this.addIgnoreFile(e, then));
124+
await Promise.all(newIg.map((e) => this.addIgnoreFile(e)));
130125
}
131126

132-
addIgnoreFile(file: Entry, then: () => void): void {
133-
const ig = path.resolve(this.path, file[0]);
134-
this.ide
135-
.readFile(ig)
136-
.then((data) => {
137-
this.onReadIgnoreFile(file, data, then);
138-
})
139-
.catch((err) => {
140-
this.emit("error", err);
141-
});
127+
async addIgnoreFile(fileEntry: Entry) {
128+
const ig = path.resolve(this.path, fileEntry[0]);
129+
130+
try {
131+
const file = await this.ide.readFile(ig);
132+
this.onReadIgnoreFile(fileEntry, file);
133+
} catch (err) {
134+
this.emit("error", err);
135+
}
142136
}
143137

144-
onReadIgnoreFile(file: Entry, data: string, then: () => void): void {
138+
onReadIgnoreFile(file: Entry, data: string): void {
145139
const mmopt = {
146140
matchBase: true,
147141
dot: true,
148142
flipNegate: true,
149143
nocase: true,
150144
};
145+
151146
const rules = data
152147
.split(/\r?\n/)
153148
.filter((line) => !/^#|^$/.test(line.trim()))
@@ -156,8 +151,6 @@ class Walker extends EventEmitter {
156151
});
157152

158153
this.ignoreRules[file[0]] = rules;
159-
160-
then();
161154
}
162155

163156
addIgnoreRules(rules: string[]) {
@@ -167,6 +160,7 @@ class Walker extends EventEmitter {
167160
flipNegate: true,
168161
nocase: true,
169162
};
163+
170164
const minimatchRules = rules
171165
.filter((line) => !/^#|^$/.test(line.trim()))
172166
.map((rule) => {
@@ -176,15 +170,22 @@ class Walker extends EventEmitter {
176170
this.ignoreRules[".defaultignore"] = minimatchRules;
177171
}
178172

179-
filterEntries(): void {
180-
const filtered = this.entries!.map((entry) => {
181-
const passFile = this.filterEntry(entry[0]);
182-
const passDir = this.filterEntry(entry[0], true);
183-
return passFile || passDir ? [entry, passFile, passDir] : false;
184-
}).filter((e) => e) as [Entry, boolean, boolean][];
173+
async *filterEntries() {
174+
const filtered = (await Promise.all(
175+
this.entries!.map(async (entry) => {
176+
const passFile = await this.filterEntry(entry[0]);
177+
const passDir = await this.filterEntry(entry[0], true);
178+
return passFile || passDir ? [entry, passFile, passDir] : false;
179+
}),
180+
).then((entries) => entries.filter((e) => e))) as [
181+
Entry,
182+
boolean,
183+
boolean,
184+
][];
185185
let entryCount = filtered.length;
186186
if (entryCount === 0) {
187187
this.emit("done", this.result);
188+
yield this.result;
188189
} else {
189190
const then = () => {
190191
if (--entryCount === 0) {
@@ -195,10 +196,12 @@ class Walker extends EventEmitter {
195196
this.emit("done", this.result);
196197
}
197198
};
198-
filtered.forEach((filt) => {
199-
const [entry, file, dir] = filt;
200-
this.stat(entry, file, dir, then);
201-
});
199+
200+
for (const [entry, file, dir] of filtered) {
201+
for await (const statResult of this.stat(entry, file, dir, then)) {
202+
yield statResult;
203+
}
204+
}
202205
}
203206
}
204207

@@ -212,29 +215,36 @@ class Walker extends EventEmitter {
212215
return entry[1] === Directory;
213216
}
214217

215-
onstat(entry: Entry, file: boolean, dir: boolean, then: () => void): void {
218+
async *onstat(entry: Entry, file: boolean, dir: boolean, then: () => void) {
216219
const abs = this.path + "/" + entry[0];
217220
const isSymbolicLink = this.entryIsSymlink(entry);
218221
if (!this.entryIsDirectory(entry)) {
219222
if (file && !this.onlyDirs) {
220223
this.result.add(abs.slice(this.root.length + 1));
221224
}
222225
then();
226+
yield this.result;
223227
} else {
224228
if (dir) {
225-
this.walker(
229+
yield* this.walker(
226230
entry[0],
227-
{ isSymbolicLink, exact: this.filterEntry(entry[0] + "/") },
231+
{ isSymbolicLink, exact: await this.filterEntry(entry[0] + "/") },
228232
then,
229233
);
230234
} else {
231235
then();
236+
yield this.result;
232237
}
233238
}
234239
}
235240

236-
stat(entry: Entry, file: boolean, dir: boolean, then: () => void): void {
237-
this.onstat(entry, file, dir, then);
241+
async *stat(
242+
entry: Entry,
243+
file: boolean,
244+
dir: boolean,
245+
then: () => void,
246+
): any {
247+
yield* this.onstat(entry, file, dir, then);
238248
}
239249

240250
walkerOpt(entry: string, opts: Partial<WalkerOptions>): WalkerOptions {
@@ -249,29 +259,36 @@ class Walker extends EventEmitter {
249259
};
250260
}
251261

252-
walker(entry: string, opts: Partial<WalkerOptions>, then: () => void): void {
253-
new Walker(this.walkerOpt(entry, opts), this.ide).on("done", then).start();
262+
async *walker(entry: string, opts: Partial<WalkerOptions>, then: () => void) {
263+
const walker = new Walker(this.walkerOpt(entry, opts), this.ide);
264+
265+
walker.on("done", then);
266+
yield* walker.start();
254267
}
255268

256-
filterEntry(
269+
async filterEntry(
257270
entry: string,
258271
partial?: boolean,
259272
entryBasename?: string,
260-
): boolean {
273+
): Promise<boolean> {
261274
let included = true;
262275

263276
if (this.parent && this.parent.filterEntry) {
264277
const parentEntry = this.basename + "/" + entry;
265278
const parentBasename = entryBasename || entry;
266-
included = this.parent.filterEntry(parentEntry, partial, parentBasename);
279+
included = await this.parent.filterEntry(
280+
parentEntry,
281+
partial,
282+
parentBasename,
283+
);
267284
if (!included && !this.exact) {
268285
return false;
269286
}
270287
}
271288

272-
this.ignoreFiles.forEach((f) => {
289+
for (const f of this.ignoreFiles) {
273290
if (this.ignoreRules[f]) {
274-
this.ignoreRules[f].forEach((rule) => {
291+
for (const rule of this.ignoreRules[f]) {
275292
if (rule.negate !== included) {
276293
const isRelativeRule =
277294
entryBasename &&
@@ -299,29 +316,14 @@ class Walker extends EventEmitter {
299316
included = rule.negate;
300317
}
301318
}
302-
});
319+
}
303320
}
304-
});
321+
}
305322

306323
return included;
307324
}
308325
}
309326

310-
interface WalkCallback {
311-
(err: Error | null, result?: string[]): void;
312-
}
313-
314-
async function walkDirWithCallback(
315-
opts: WalkerOptions,
316-
ide: IDE,
317-
callback?: WalkCallback,
318-
): Promise<string[] | void> {
319-
const p = new Promise<string[]>((resolve, reject) => {
320-
new Walker(opts, ide).on("done", resolve).on("error", reject).start();
321-
});
322-
return callback ? p.then((res) => callback(null, res), callback) : p;
323-
}
324-
325327
const defaultOptions: WalkerOptions = {
326328
ignoreFiles: [".gitignore", ".continueignore"],
327329
onlyDirs: false,
@@ -333,40 +335,41 @@ export async function walkDir(
333335
ide: IDE,
334336
_options?: WalkerOptions,
335337
): Promise<string[]> {
338+
let entries: string[] = [];
336339
const options = { ...defaultOptions, ..._options };
337-
return new Promise((resolve, reject) => {
338-
walkDirWithCallback(
339-
{
340-
path,
341-
ignoreFiles: options.ignoreFiles,
342-
onlyDirs: options.onlyDirs,
343-
follow: true,
344-
includeEmpty: false,
345-
additionalIgnoreRules: options.additionalIgnoreRules,
346-
},
347-
ide,
348-
async (err, result) => {
349-
if (err) {
350-
reject(err);
351-
} else {
352-
const relativePaths = result || [];
353-
if (options?.returnRelativePaths) {
354-
resolve(relativePaths);
355-
} else {
356-
const pathSep = await ide.pathSep();
357-
if (pathSep === "/") {
358-
resolve(relativePaths.map((p) => path + pathSep + p));
359-
} else {
360-
// Need to replace with windows path sep
361-
resolve(
362-
relativePaths.map(
363-
(p) => path + pathSep + p.split("/").join(pathSep),
364-
),
365-
);
366-
}
367-
}
368-
}
369-
},
370-
);
371-
});
340+
341+
const walker = new Walker(
342+
{
343+
path,
344+
ignoreFiles: options.ignoreFiles,
345+
onlyDirs: options.onlyDirs,
346+
follow: true,
347+
includeEmpty: false,
348+
additionalIgnoreRules: options.additionalIgnoreRules,
349+
},
350+
ide,
351+
);
352+
353+
try {
354+
for await (const walkedEntries of walker.start()) {
355+
entries = [...walkedEntries];
356+
}
357+
} catch (err) {
358+
console.error(`Error walking directories: ${err}`);
359+
throw err;
360+
}
361+
362+
const relativePaths = entries || [];
363+
364+
if (options?.returnRelativePaths) {
365+
return relativePaths;
366+
}
367+
368+
const pathSep = await ide.pathSep();
369+
370+
if (pathSep === "/") {
371+
return relativePaths.map((p) => path + pathSep + p);
372+
}
373+
374+
return relativePaths.map((p) => path + pathSep + p.split("/").join(pathSep));
372375
}

core/test/walkDir.test.ts

+1-1
Original file line numberDiff line numberDiff line change
@@ -264,7 +264,7 @@ describe("walkDir", () => {
264264
});
265265

266266
test("should walk continue/extensions/vscode without getting any files in the .continueignore", async () => {
267-
const vscodePath = path.join(__dirname, "..", "extensions", "vscode");
267+
const vscodePath = path.join(__dirname, "../..", "extensions", "vscode");
268268
const results = await walkDir(vscodePath, ide, {
269269
ignoreFiles: [".gitignore", ".continueignore"],
270270
});

extensions/vscode/package.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "continue",
33
"icon": "media/icon.png",
4-
"version": "0.9.185",
4+
"version": "0.9.186",
55
"repository": {
66
"type": "git",
77
"url": "https://github.com/continuedev/continue"

0 commit comments

Comments
 (0)