Legion RAG: Performance That Doesn’t Break the Bank
Published on Oct 30, 2024
Lead Research Scientist and Founder
Senior AI Researcher
Legion’s AI summary
You'll learn how Legion's Retrieval Augmented Generation (RAG) system delivers high performance at a fraction of the cost.
- Dynamic chunking and embedding: Legion's novel approach ensures superior retrieval while keeping costs low
- Contextual Retrieval comparison: See how Legion's RAG system stacks up against Anthropic's technique on the Codebases dataset
- Detailed cost analysis: Explore the significant cost savings of Legion's ingestion method compared to Contextual Retrieval
Read on to uncover the power and efficiency of Legion's RAG platform for enterprise AI assistants.
All techniques mentioned, barring Legion RAG, use Voyager-Large-2 as their embedding model and Claude-2 for chunk contextualization. Approximate Cost evaluated for Large Enterprises with 1000+ employees housing ~2 million documents (detailed cost analysis in section below).
Figure credits: Alison Martin, Legion
At Legion, we build AI Agents for enterprises that generate answers based on proprietary knowledge bases while respecting access controls. We do this using a powerful Retrieval Augmented Generation (RAG) system to support natural language queries.
RAG enables Large Language Models (LLMs) to answer questions about a user’s private knowledge base without training these models on private data. In a typical RAG system, the knowledge base is chunked into smaller pieces, each embedded into a vector space using an embedding model, allowing for relevant chunk retrieval for a given query.
Challenges include ensuring context when chunking. For example, if two sentences are split, disambiguation may be impossible without additional context. Techniques have been developed to address such issues, like generic document summaries to chunks and summary-based indexing.
Anthropic introduced Contextual Retrieval to enhance chunk context using the overall document.
In this post, we:
- Evaluate Legion’s chunking and embedding strategy against Anthropic’s Contextual Retrieval on the Codebases dataset and show comparable performance (~93% recall@5).
- Share a detailed cost analysis showing Legion's method is 1/300th the cost of Contextual Retrieval.
- Highlight nuances when using new benchmarking datasets to evaluate RAG platforms.
Legion’s RAG and Contextual Retrieval: Examining the Approaches
Details on the Codebases dataset:
- Documents: 90 code files in languages like Python, Rust, C++ from various repositories.
- User queries: 248 queries.
- Golden chunks: Provided for each query, necessary parts of the code files to answer them.
Evaluating Legion:
Our platform ingests the 90 documents, uses custom chunking for superior retrieval performance, and collects top-5 chunks per query.
We use a two-step evaluation: edit-distance between top-5 chunks and golden chunks, and human-in-the-loop evaluation for assessing relevance.
Our Results:
- Edit-distance evaluation: 55% of the top-5 chunks aligned closely with golden chunks.
- Human evaluation: Identified issues with golden chunks and query quality.
Final Results:
While comparing methods, Legion RAG achieves 93.2% recall in top-5 retrieval, comparable to Contextual Retrieval, emphasizing cost-effectiveness for enterprises.
Ingestion cost comparison: Legion RAG and Contextual Retrieval
Cost analysis focuses on deployment, API calls, and model running costs. Legion RAG’s ingestion is significantly cheaper:
| Enterprise Size | Legion RAG Cost | Contextual Retrieval Cost |
|---|---|---|
| Medium (100-1k) | ~$360 | ~$109k |
The analysis shows that while Contextual Retrieval can be costly, Legion’s method is efficient and budget-friendly.
Takeaways
- Legion’s system performs comparably to new techniques with lower costs.
- Evaluate nuances in chunk alignment and quality of queries.
- Contextual Retrieval may improve performance but at a premium cost.
- Legion's method offers competitive performance at significantly lower costs for ongoing document ingestion.
For more recent work from Legion, see Why Retrieval Agents Fail: It's Not Just the Model.
Appendix
Codebases dataset Statistics
Statistical significance of performance scores
- P value: 0.7544, indicating no significant difference.
Details of cost analysis
Contextual Retrieval:
- Long context model (Claude)
- Embedding model (Voyager-2)
Legion RAG system hosted on-premise:
- Focus on GPU cost and performance.
As enterprises scale, Contextual Retrieval costs dramatically increase.