161 lines
4.6 KiB
Markdown
161 lines
4.6 KiB
Markdown
# Advanced Retrieval Strategies
|
|
|
|
## Overview
|
|
Different retrieval approaches for finding relevant documents in RAG systems, each with specific strengths and use cases.
|
|
|
|
## Retrieval Approaches
|
|
|
|
### 1. Dense Retrieval
|
|
**Method**: Semantic similarity via embeddings
|
|
**Use Case**: Understanding meaning and context
|
|
**Example**: Finding documents about "machine learning" when query is "AI algorithms"
|
|
|
|
```python
|
|
from langchain.vectorstores import Chroma
|
|
|
|
vectorstore = Chroma.from_documents(chunks, embeddings)
|
|
results = vectorstore.similarity_search("query", k=5)
|
|
```
|
|
|
|
### 2. Sparse Retrieval
|
|
**Method**: Keyword matching (BM25, TF-IDF)
|
|
**Use Case**: Exact term matching and keyword-specific queries
|
|
**Example**: Finding documents containing specific technical terms
|
|
|
|
```python
|
|
from langchain.retrievers import BM25Retriever
|
|
|
|
bm25_retriever = BM25Retriever.from_documents(chunks)
|
|
bm25_retriever.k = 5
|
|
results = bm25_retriever.get_relevant_documents("query")
|
|
```
|
|
|
|
### 3. Hybrid Search
|
|
**Method**: Combine dense + sparse retrieval
|
|
**Use Case**: Balance between semantic understanding and keyword matching
|
|
|
|
```python
|
|
from langchain.retrievers import BM25Retriever, EnsembleRetriever
|
|
|
|
# Sparse retriever (BM25)
|
|
bm25_retriever = BM25Retriever.from_documents(chunks)
|
|
bm25_retriever.k = 5
|
|
|
|
# Dense retriever (embeddings)
|
|
embedding_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})
|
|
|
|
# Combine with weights
|
|
ensemble_retriever = EnsembleRetriever(
|
|
retrievers=[bm25_retriever, embedding_retriever],
|
|
weights=[0.3, 0.7]
|
|
)
|
|
```
|
|
|
|
### 4. Multi-Query Retrieval
|
|
**Method**: Generate multiple query variations
|
|
**Use Case**: Complex queries that can be interpreted in multiple ways
|
|
|
|
```python
|
|
from langchain.retrievers.multi_query import MultiQueryRetriever
|
|
|
|
# Generate multiple query perspectives
|
|
retriever = MultiQueryRetriever.from_llm(
|
|
retriever=vectorstore.as_retriever(),
|
|
llm=OpenAI()
|
|
)
|
|
|
|
# Single query → multiple variations → combined results
|
|
results = retriever.get_relevant_documents("What is the main topic?")
|
|
```
|
|
|
|
### 5. HyDE (Hypothetical Document Embeddings)
|
|
**Method**: Generate hypothetical documents for better retrieval
|
|
**Use Case**: When queries are very different from document style
|
|
|
|
```python
|
|
# Generate hypothetical document based on query
|
|
hypothetical_doc = llm.generate(f"Write a document about: {query}")
|
|
# Use hypothetical doc for retrieval
|
|
results = vectorstore.similarity_search(hypothetical_doc, k=5)
|
|
```
|
|
|
|
## Advanced Retrieval Patterns
|
|
|
|
### Contextual Compression
|
|
Compress retrieved documents to only include relevant parts
|
|
|
|
```python
|
|
from langchain.retrievers import ContextualCompressionRetriever
|
|
from langchain.retrievers.document_compressors import LLMChainExtractor
|
|
|
|
compressor = LLMChainExtractor.from_llm(llm)
|
|
compression_retriever = ContextualCompressionRetriever(
|
|
base_compressor=compressor,
|
|
base_retriever=vectorstore.as_retriever()
|
|
)
|
|
```
|
|
|
|
### Parent Document Retriever
|
|
Store small chunks for retrieval, return larger chunks for context
|
|
|
|
```python
|
|
from langchain.retrievers import ParentDocumentRetriever
|
|
from langchain.storage import InMemoryStore
|
|
|
|
store = InMemoryStore()
|
|
child_splitter = RecursiveCharacterTextSplitter(chunk_size=400)
|
|
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
|
|
|
|
retriever = ParentDocumentRetriever(
|
|
vectorstore=vectorstore,
|
|
docstore=store,
|
|
child_splitter=child_splitter,
|
|
parent_splitter=parent_splitter
|
|
)
|
|
```
|
|
|
|
## Retrieval Optimization Techniques
|
|
|
|
### 1. Metadata Filtering
|
|
Filter results based on document metadata
|
|
|
|
```python
|
|
results = vectorstore.similarity_search(
|
|
"query",
|
|
filter={"category": "technical", "date": {"$gte": "2023-01-01"}},
|
|
k=5
|
|
)
|
|
```
|
|
|
|
### 2. Maximal Marginal Relevance (MMR)
|
|
Balance relevance with diversity
|
|
|
|
```python
|
|
results = vectorstore.max_marginal_relevance_search(
|
|
"query",
|
|
k=5,
|
|
fetch_k=20,
|
|
lambda_mult=0.5 # 0=max diversity, 1=max relevance
|
|
)
|
|
```
|
|
|
|
### 3. Reranking
|
|
Improve top results with cross-encoder
|
|
|
|
```python
|
|
from sentence_transformers import CrossEncoder
|
|
|
|
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
|
|
candidates = vectorstore.similarity_search("query", k=20)
|
|
pairs = [[query, doc.page_content] for doc in candidates]
|
|
scores = reranker.predict(pairs)
|
|
reranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)[:5]
|
|
```
|
|
|
|
## Selection Guidelines
|
|
|
|
1. **Query Type**: Choose strategy based on typical query patterns
|
|
2. **Document Type**: Consider document structure and content
|
|
3. **Performance Requirements**: Balance quality vs speed
|
|
4. **Domain Knowledge**: Leverage domain-specific patterns
|
|
5. **User Expectations**: Match retrieval behavior to user expectations |