Adding context files recursively from a directory

[hartikainen]

Thanks for such a great and pleasant-to-use package!

I've had several occasions recently where I would've wanted to add the contents of a whole directory to the context of my model. I can do that manually but with more than a couple of files that becomes quite clumsy. It would be great if there was a function like gptel-add-directory or similar, which would allow files in the directory to be added, ideally recursively.

[metachip]

Hi @hartikainen

This is a great idea.

It is related to a wider discussion about how to manage context beyond the "labour-intensive mechanism" that it offers now. Especially if one is used to using markdown or org mode as the source syntax for dialogue with the models. My view is that the management of context within gptel is an area that can be improved significantly with what are, conceptually at least, relatively minor changes.

The ability to add directories of files could be difficult to "keep simple". All nested subdirectories? Only files with specific extensions? Which ones? How should they be filtered? I'm sure there are ways to simplify this using metaphors like "dired", which can already be used in a way similar to this, But it is not automated in the way I think you are suggesting.

Have a look at the summary below and the related issues on this topic.

See what you think :)

1. Context Buffer Navigation (#459)

Suggestion:

• Make context previews collapsible in the context buffer
• Hide previews by default with TAB to reveal (similar to Magit)
• Current preview chunks can be unwieldy when large

Status: Under consideration, but no specific implementation committed yet.

2. Multiple Context Sets (#475, #459)

Suggestion:

• Support multiple named context buffers
• Allow association between context sets and chat buffers
• Use prefix arguments (C-u) to select context sets
• Add widget to chat buffers showing active context

Karthink's Response (#475):

• Concerned about user confusion and complexity
• Prefers one-shot interactions without additional decisions
• Raises questions about ambiguity in context assignment
• Notes problems with maintaining internal registry
• Suggests project-local contexts might be more intuitive

3. Links as Context (#481)

Suggestion:

• Allow links in non-chat buffers
• Support text file content via links
• Remove standalone link requirement
• Use :CONTEXT: drawer in org-mode to isolate LLM links
• Support region targeting via link labels

Karthink's Response:

• Supportive of using links for per-buffer context
• Concerned about accidental inclusion of links
• Prefers standalone links for clear intent
• Willing to add text file support
• Cautious about org-specific link types

4. Quick Context Management (#486)

Suggestion:

• Add command to quickly delete all context
• Provide way to temporarily ignore context

Status: Implemented

• Context deletion added to transient menu (commit a844612)
• Context ignoring available via -i option in transient menu
• Can be toggled per-buffer using gptel-use-context

5. Multi-Modal Support

Status: Already Implemented

• Image and media support added before 0.9.6
• PDF support added for Claude 3.5 Sonnet
• Can use gptel-add or links in Org/Markdown chat buffers

Current Direction

The discussion seems to be moving towards:

1. Using links as the primary mechanism for context management (Media (and other) links. #481)
2. Maintaining simplicity in the interface
3. Avoiding complex context management systems
4. Providing better ergonomics for existing features

Karthink appears most receptive to improvements that:

• Maintain simplicity
• Avoid additional user decisions
• Provide clear intent
• Work within existing Emacs paradigms

The most promising avenue appears to be the links-based approach (#481), which could provide a more transparent and user-friendly way to manage context while avoiding the complexity of multiple context sets.

[2024-12-22 Sun 10:03]

[karthink]

@hartikainen see PR #438, which is almost ready to be merged.

Also, as a fallback for now you can mark all files in a directory wilth t in dired and add with gptel-add.

[karthink]

@metachip Thanks for the summary.

Hide previews by default with TAB to reveal (similar to Magit)

Current preview chunks can be unwieldy when large

Status: Under consideration, but no specific implementation committed yet.

This is trivial to add if we add magit-section as a dependency of gptel, I'm just hesitant to do that. gptel has one dependency (transient), and updates to it have cause gptel to break in minor ways multiple times this year.

3. Links as Context (Media (and other) links. #481)

Yup, this is going to happen, I just have to merge the tool-use branch first.

[collinarnett]

It would be really cool if you could exclude based on a .gitignore if in a git project.

[benthamite]

@collinarnett, @karthink: Can you describe the intended behavior? Do we want to call gptel-add with point on the directory of a repository and add all files recursively minus the files in .gitignore? Do we also want to exclude those files if gptel-add is called with point on a subdirectory of the repo? What if one or more of the ignored files are explicitly selected (as opposed to being in one of the selected directories)? And how should we handle situations when the user does want to add the ignored files?

[collinarnett]

@benthamite

I use lsp-mode for my lsp support in emacs and whenever I activate lsp-mode on a new project I get a dialog that looks like this.

Something similar would be useful for gptel to discover a .gitignore. Next when I select a directory with gpt-add it will recursively glob all the files as long as they do not match any patterns in .gitignore.

My motivation for wanting the recursive behavior is that I don't have projects that are very large so the entire project can easily fit in context. Ideally I don't even need to use gpt-add and my entire repository is added to context when I open a file but that's just me. I just need the ability to exclude things like node-modules and .git etc.

[karthink]

Not a feature I myself would use since I only ever add single files, but here's what I'm picturing:

Do we want to call gptel-add with point on the directory of a repository and add all files recursively minus the files in .gitignore?

In dired mode, yes.

Do we also want to exclude those files if gptel-add is called with point on a subdirectory of the repo?

Yes, you'd have to use (locate-dominating-file "." ".gitignore") whenever adding any file.

What if one or more of the ignored files are explicitly selected (as opposed to being in one of the selected directories)?

This case probably requires another y-or-n-p prompt, asking the user if they want to add selected, gitignored files.

And how should we handle situations when the user does want to add the ignored files?

I'm not sure yet.

---

On the whole this opens up a bundle of edge cases. While I don't mind adding this feature, more generally I'm not convinced that sending whole directories with every gptel request is a good idea in the first place. This is quite expensive and wasteful.

A better solution to the context problem -- and what users really want, I think -- is for large context to be provided by a RAG implementation for gptel, either via a plug-in or as part of gptel. Implementing one in gptel is a complicated project -- at minimum gptel will have to maintain an sqlite database per project, and depend on non-mature external projects like sqlite-vec.

[hartikainen]

Thanks for such detailed and thoughtful responses! Here are my two cents.

• Make context previews collapsible in the context buffer
• Hide previews by default with TAB to reveal (similar to Magit)
• Current preview chunks can be unwieldy when large

I'm personally quite comfortable handling even large contexts (e.g. diffs) using Magit's collapsible views. The magit-section-show-level-{1,2,3,4} functionality is particularly useful here. Similar functionality in gptel would take me long way, but I do understand @karthink's hesitation of bringing extra dependencies into the project.

see PR #438, which is almost ready to be merged.

I think this would already be a huge improvement for me. Thanks a ton, @benthamite, for contributing this!

3. Links as Context (#481)

Links seem like an interesting approach, although I can't comment yet how they fit my workflow. I'll definitely try them out when they're released.

It would be really cool if you could exclude based on a .gitignore if in a git project.

I too think that having an option to do this would be great! @karthink's picture of how this would work fits my current use cases well.

I'm not convinced that sending whole directories with every gptel request is a good idea

A better solution to the context problem -- and what users really want, I think -- is for large context to be provided by a RAG implementation for gptel

My use cases at the moment are simple enough that I don't mind carrying O(10) small files as the context. Having access to some sort of RAG setup would definitely open up interesting new use cases, although I agree that implementing such would be quite non-trivial.

Thanks everyone for the input. It's super exciting to see gptel evolving 🙂

[benthamite]

The PR is now merged. We decided not to add support for .gitignore files because that raises some issues that were beyond the scope of the PR as initially conceived. I may try to add this functionality in the future, but since I don’t use lsp-mode or other packages that operate on projects, I don’t have very clear intuitions about how to handle various edge cases, so others may be in a better position to implement it.

[karthink]

Please open a new feature request if you wish to pursue .gitignore support in gptel-context.

[benthamite]

Hello, here is a preliminary attempt to provide support for .gitignore files. When you call gptel-add in Dired with point on a directory containing a Git repository with a .gitignore file, it should skip any files or directories listed there. If anyone decides to test it, please let me know if it works as expected. Thanks!