feat: custom web RAG with Firecrawl

[justinh-rahb]

Firecrawl is highly suitable for custom web Retrieval-Augmented Generation (RAG) pipelines due to its advanced features and flexibility. Here are the key highlights:

1. Smart LLM Scraping: Converts websites into clean, LLM-ready markdown, ensuring data is optimized for language models which improves the accuracy and relevance of RAG outputs.

2. Comprehensive Capabilities: Efficiently handles JavaScript-heavy sites and complex site navigations for thorough data retrieval, critical for feeding accurate information into RAG systems.

3. Self-Hostable and Scalable: Offers both a hosted version and a self-hostable option, giving you control over deployment and scalability, which is ideal for handling large-scale data needs in RAG pipelines.

4. Efficient and Cost-Effective: Noted for significant savings in computational costs and time, enhancing operational efficiency in large-scale deployments.

This combination of smart scraping, comprehensive data handling, scalability, and cost-efficiency makes Firecrawl an excellent choice for enhancing custom RAG pipelines
Firecrawl.dev | Introduction | Scrape Docs

[tjbck]

I guess the issue here is knowing when to search the web, one solution here is asking the users to prefix their prompt with a designated command (e.g. !search what's open webui?)

[bannert1337]

Why not as a tool the LLM can call whenever it wants/needs it.

[atgehrhardt]

One issue I can think of with it being a tool is that it's going to be really hit or miss. This would be so useful, that while a tool implementation would be cool, smaller models would really struggle, so I think more people would get benefit out of being able to invoke it explicitly.

It can always be implemented both ways, but while the tool would be cooler, it also would be less applicable across the board.