Embeddings & Vector Stores
Transforming text into semantic search - the foundation of modern AI applications
Understanding Embeddings
Embeddings are one of the most powerful concepts in modern AI. They transform text, images, or other data into numerical vectors that capture semantic meaning. This transformation unlocks capabilities like semantic search, recommendation systems, and retrieval-augmented generation.
What Are Embeddings?
An embedding is a dense vector representation of data that captures its semantic meaning in a high-dimensional space. Instead of treating words or sentences as discrete symbols, embeddings represent them as arrays of numbers.
Simple Example
Real embeddings are typically 384, 768, 1536, or even 3072 dimensions!
Semantic Capture
Similar meanings produce similar vectors, even with different words
Dense Representation
Every dimension carries meaningful information about the text
Searchable
Mathematical operations enable similarity comparisons
Why Embeddings Matter for Finance
Traditional Keyword Search vs Semantic Search
Traditional search matches exact keywords. If you search for "revenue growth" but the document says "increasing sales", you miss it. Embeddings understand that these phrases are semantically related.
Keyword Search
Semantic Search (Embeddings)
How Embeddings Work: The Intuition
Think of embeddings as coordinates in a multi-dimensional "meaning space". Words or phrases with similar meanings are located close together in this space.
Visualizing Embeddings (2D Simplification)
Hover over dots to see the phrases. Notice how semantically similar phrases cluster together!
Key Insight
In real embeddings, we have hundreds or thousands of dimensions, not just 2. This high-dimensional space allows capturing incredibly nuanced semantic relationships. "Revenue growth" and "increasing sales" will have vectors that point in nearly the same direction in this space.
Popular Embedding Models
OpenAI Embeddings
ProprietaryCohere Embed
ProprietarySentence Transformers
Open SourceVoyage AI
ProprietaryFinance-Specific Recommendation
For financial applications, consider Voyage AI's finance-2 modelwhich is specifically trained on financial documents, or OpenAI's text-embedding-3-large for general-purpose high-quality embeddings. For cost-sensitive or privacy-critical applications, self-hosted Sentence Transformers are excellent choices.
How to Generate Embeddings
Using OpenAI API
from openai import OpenAI
client = OpenAI(api_key="your-api-key")
# Generate embedding for financial text
response = client.embeddings.create(
model="text-embedding-3-small",
input="The company reported strong quarterly earnings with 15% revenue growth"
)
embedding = response.data[0].embedding
print(f"Embedding dimensions: {len(embedding)}")
print(f"First 5 values: {embedding[:5]}")
# Output:
# Embedding dimensions: 1536
# First 5 values: [0.0023, -0.0045, 0.0067, 0.0012, -0.0089]Using Sentence Transformers (Self-Hosted)
from sentence_transformers import SentenceTransformer
# Load model (downloads on first run)
model = SentenceTransformer('all-MiniLM-L6-v2')
# Generate embeddings
texts = [
"The company's revenue increased by 15%",
"Sales growth exceeded expectations",
"Credit risk remains elevated"
]
embeddings = model.encode(texts)
print(f"Shape: {embeddings.shape}")
print(f"Embedding for text 1: {embeddings[0][:5]}")
# Output:
# Shape: (3, 384)
# Embedding for text 1: [0.023, -0.045, 0.067, 0.012, -0.089]Key Considerations
- โขConsistency: Always use the same model for both indexing and querying
- โขChunking: Break long documents into smaller chunks before embedding
- โขBatch Processing: Generate embeddings in batches for efficiency
- โขCaching: Store embeddings once generated to avoid recomputation
From Text to Similarity Search
Now that we understand embeddings, let's see how they power semantic search. The process involves three key steps: encoding text into vectors, measuring similarity between vectors, and retrieving the most relevant results.
The Complete Workflow
Indexing Phase: Embed Your Documents
Before you can search, you need to convert all your documents into embeddings and store them in a vector database.
Query Phase: Embed Your Question
When a user asks a question, convert it to an embedding using the same model you used for indexing.
Similarity Calculation: Find Nearest Neighbors
Calculate the similarity between the query embedding and all document embeddings. Return the top matches.
Measuring Similarity: Distance Metrics
How do we actually measure if two vectors are "similar"? There are several mathematical approaches, each with different properties.
Cosine Similarity
Range: [-1, 1]
1 = identical
0 = orthogonal
-1 = opposite
Measures the angle between vectors, ignoring magnitude.
Euclidean Distance
Range: [0, โ]
0 = identical
โ = very different
Straight-line distance between two points in space.
Dot Product
Range: [-โ, โ]
Higher = more similar
Simple multiplication and sum. Fast to compute.
Recommendation for Finance
Cosine similarity is the standard choice for text embeddings and is what most embedding models are optimized for. It works well even when documents have different lengths (which is common in financial documents like 10-Ks, earnings calls, and research reports).
Practical Example: Financial Document Search
Let's walk through a complete example of semantic search on financial documents.
Complete Python Implementation
import numpy as np
from openai import OpenAI
from sklearn.metrics.pairwise import cosine_similarity
client = OpenAI(api_key="your-api-key")
# Sample financial documents
documents = [
"The company's Q4 revenue increased by 15% year-over-year to $2.3B",
"Credit default rates rose to 2.8% in the commercial lending portfolio",
"Net interest margin compressed by 25 basis points due to rate changes",
"Market volatility led to a 12% decline in trading revenue",
"The firm announced a $500M share buyback program"
]
def get_embedding(text):
"""Get embedding for a single text"""
response = client.embeddings.create(
model="text-embedding-3-small",
input=text
)
return response.data[0].embedding
# Step 1: Create embeddings for all documents (Indexing)
print("Creating document embeddings...")
doc_embeddings = [get_embedding(doc) for doc in documents]
# Step 2: User asks a question (Query)
query = "How is the company performing in terms of revenue?"
print(f"\nQuery: {query}")
# Step 3: Embed the query
query_embedding = get_embedding(query)
# Step 4: Calculate similarities
query_emb_array = np.array(query_embedding).reshape(1, -1)
doc_emb_array = np.array(doc_embeddings)
similarities = cosine_similarity(query_emb_array, doc_emb_array)[0]
# Step 5: Get top results
top_indices = np.argsort(similarities)[::-1][:3]
print("\nTop 3 Results:")
for i, idx in enumerate(top_indices, 1):
print(f"{i}. [Score: {similarities[idx]:.3f}] {documents[idx]}")
# Output:
# Top 3 Results:
# 1. [Score: 0.847] The company's Q4 revenue increased by 15% year-over-year to $2.3B
# 2. [Score: 0.512] Market volatility led to a 12% decline in trading revenue
# 3. [Score: 0.432] Net interest margin compressed by 25 basis points due to rate changesWhy This Works
Limitations of Simple In-Memory Search
Vector Databases: Built for Scale
Vector databases are specialized databases designed to store, index, and query high-dimensional vector embeddings efficiently. They solve the scalability and performance problems of naive in-memory similarity search.
Why Not Just Use a Regular Database?
Traditional Databases (PostgreSQL, MySQL)
- โขOptimized for exact matches, not similarity
- โขNo specialized indexing for high-dimensional vectors
- โขSlow for computing distances across millions of vectors
- โขCan't efficiently handle 1536-dimensional vectors
Vector Databases
- โขBuilt for similarity search (ANN - Approximate Nearest Neighbors)
- โขSpecialized indexes: HNSW, IVF, LSH for fast search
- โขSub-second queries on millions/billions of vectors
- โขNative support for high-dimensional embeddings
Key Insight: Approximate vs Exact Search
Vector databases use Approximate Nearest Neighbor (ANN) algorithms. Instead of comparing against every single vector (exact but slow), they use clever indexing to find the approximately nearest neighbors in milliseconds. For most applications, 95-99% accuracy is more than sufficient and worth the massive speed gains.
Popular Vector Database Options
Pinecone
Fully managed, cloud-native vector database
Pros
- โFully managed, no infrastructure to maintain
- โAuto-scaling and high availability
- โExcellent performance and low latency
- โSimple API and great documentation
- โMetadata filtering and hybrid search
Cons
- โProprietary, vendor lock-in
- โCan get expensive at scale
- โNo self-hosted option
- โData stored in their cloud
Weaviate
Open-source vector database with GraphQL API
Pros
- โOpen source, self-hostable
- โGraphQL API for flexible queries
- โBuilt-in vectorization modules
- โRich filtering and schema support
- โManaged cloud option available
Cons
- โRequires infrastructure management if self-hosted
- โGraphQL may be overkill for simple use cases
- โLearning curve for advanced features
Chroma
The AI-native open-source embedding database
Pros
- โExtremely simple to get started
- โRuns embedded in Python (no server needed)
- โBuilt-in embedding functions
- โPerfect for prototyping and development
- โFree and open source
Cons
- โNot designed for large-scale production
- โLimited scalability compared to others
- โFewer enterprise features
FAISS (Facebook AI Similarity Search)
High-performance library for similarity search
Pros
- โExtremely fast, optimized for GPUs
- โIndustry-proven, used at Meta scale
- โMultiple indexing algorithms (IVF, HNSW)
- โNo external dependencies
- โFree and open source (MIT license)
Cons
- โNot a full database, just a library
- โNo built-in persistence or metadata
- โYou handle all infrastructure yourself
- โLower-level API, more code required
pgvector (PostgreSQL Extension)
Add vector similarity search to PostgreSQL
Pros
- โUse existing PostgreSQL infrastructure
- โCombines structured + vector data in one DB
- โACID transactions and PostgreSQL features
- โFamiliar SQL interface
- โFree and open source
Cons
- โSlower than specialized vector DBs at scale
- โLimited to PostgreSQL ecosystem
- โLess optimized for pure vector workloads
Choosing the Right Vector Database
The choice depends on your specific requirements. Consider these key factors:
Managed (Pinecone): Zero ops, pay for convenience
Self-hosted (Weaviate, Chroma, FAISS): More control, compliance-friendly
Ultra-fast (FAISS, Pinecone): Best for high-throughput
Balanced (Weaviate, Chroma): Good for most use cases
Hybrid search: Weaviate excels here
Simplicity: Chroma for quick prototypes
Advanced filtering: Pinecone, Weaviate
For regulated industries, self-hosted options (Weaviate, FAISS, pgvector) may be required for data residency and audit requirements.
Recommendations for Financial Services
RAG: Retrieval-Augmented Generation
RAG is the most important architecture pattern in production Gen AI systems today. It combines the power of foundation models with your own knowledge base to produce accurate, grounded, and up-to-date responses.
What is RAG?
Retrieval-Augmented Generation (RAG) is a technique that enhances LLM responses by retrieving relevant information from a knowledge base before generating an answer. Instead of relying solely on the model's training data, RAG gives the model access to your current, specific information.
The RAG Workflow (Simplified)
RAG vs Fine-Tuning vs Baseline LLM
Baseline LLM
Fine-Tuning
RAG
When to Use RAG in Finance
RAG is ideal for financial applications where information changes frequently (market data, regulations, company financials) and accuracy is critical. Use RAG for: document Q&A, compliance chatbots, research assistants, and any scenario requiring citations and grounding.
Complete RAG Architecture
A production RAG system has two phases: an indexing phase (done once or periodically) and a query phase (done on every user request).
Phase 1: Indexing (Offline)
Phase 2: Query (Online/Real-time)
Step 1: Document Loading & Extraction
The first step in building a RAG system is loading and extracting content from your documents. Financial documents come in various formats (PDFs, Word docs, HTML, databases) and contain different types of content (text, tables, images). Choosing the right extraction tool is critical.
Popular Document Loading Libraries
PyPDF / PyPDF2
SimpleBasic PDF text extraction. Fast and lightweight, but struggles with complex layouts, tables, and scanned documents.
from pypdf import PdfReader
reader = PdfReader("apple_10k.pdf")
text = ""
for page in reader.pages:
text += page.extract_text()
print(f"Extracted {len(text)} characters")PDFPlumber
RecommendedBetter table extraction and layout awareness. Excellent for financial documents with structured data like balance sheets and income statements.
import pdfplumber
with pdfplumber.open("apple_10k.pdf") as pdf:
for page in pdf.pages:
# Extract text
text = page.extract_text()
# Extract tables
tables = page.extract_tables()
for table in tables:
# Table is a list of lists (rows and columns)
print(f"Found table with {len(table)} rows")
# Convert to pandas DataFrame for processing
df = pd.DataFrame(table[1:], columns=table[0])Unstructured.io
Production-ReadyHandles multiple formats (PDF, DOCX, HTML, images) with advanced layout detection. Preserves document structure and metadata. Great for production pipelines.
from unstructured.partition.auto import partition
# Auto-detects format and extracts intelligently
elements = partition("apple_10k.pdf")
for element in elements:
print(f"Type: {element.category}")
print(f"Text: {element.text}")
print(f"Metadata: {element.metadata}")
# Categories: Title, NarrativeText, Table, Image, etc.LlamaParse (LlamaIndex)
AI-PoweredUses LLMs to intelligently parse complex documents. Best-in-class for challenging layouts, multi-column text, and complex tables. Paid service.
from llama_parse import LlamaParse
parser = LlamaParse(
api_key="your-key",
result_type="markdown" # or "text"
)
documents = parser.load_data("apple_10k.pdf")
# Returns well-formatted markdown with preserved structure
for doc in documents:
print(doc.text) # Clean, structured outputHandling Different Content Types
๐ Text Content
Most straightforward. Extract and preserve paragraph structure. Watch out for headers/footers, page numbers, and watermarks that add noise.
๐ Tables
Critical in financial docs but challenging for RAG. Tables need special handling because embeddings don't naturally understand tabular structure.
๐ผ๏ธ Images & Charts
Graphs, charts, diagrams contain valuable insights but require OCR or vision models to extract.
Challenges with Financial Documents
Step 2: Understanding Tokens & Tokenization
Before chunking, you need to understand tokens - the fundamental units that LLMs and embedding models work with. Token counts directly impact chunking, context windows, and costs.
What Are Tokens?
Tokens are subword units that models use to process text. They're not exactly words - a single word can be one or multiple tokens depending on its length and frequency.
Different Models, Different Tokenizers
Each model family has its own tokenizer. The same text will have different token counts depending on which tokenizer you use.
OpenAI (tiktoken)
GPT-4, GPT-3.5import tiktoken
# For GPT-4, GPT-3.5-turbo
enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("The company's revenue increased")
print(f"Tokens: {len(tokens)}") # 5 tokensHugging Face
Llama, Mistral, etc.from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b")
tokens = tokenizer.encode("The company's revenue increased")
print(f"Tokens: {len(tokens)}") # May differ!Should You Tokenize Before Chunking?
Option 1: Tokenize First
Convert text to tokens, split by token count, then decode back to text for embedding.
Option 2: Estimate Tokens (Recommended)
Split by characters/words, estimate token count using the 4 chars โ 1 token rule.
Use character-based chunking with token estimation. Most libraries (LangChain, LlamaIndex) use this approach. Split by characters (e.g., 2000 chars โ 500 tokens), respect sentence boundaries, and verify token counts don't exceed limits.
Token Counting in Practice
Always verify your chunks fit within model limits:
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4")
# Check chunk token count before embedding
chunk = "Your document chunk text here..."
token_count = len(enc.encode(chunk))
# OpenAI text-embedding-3-small limit: 8191 tokens
if token_count > 8000: # Leave buffer
print(f"Warning: Chunk has {token_count} tokens, may need splitting")
# Also important: LLM context window management
# When building RAG prompt: retrieved chunks + prompt < context limit
retrieved_tokens = sum(len(enc.encode(chunk)) for chunk in retrieved_chunks)
prompt_tokens = len(enc.encode(system_prompt + user_query))
total_tokens = retrieved_tokens + prompt_tokens
if total_tokens > 100000: # Example: GPT-4 Turbo limit
print("Warning: May exceed context window")Step 3: Chunking Strategies
Now that you have extracted text and understand tokens, it's time to split your documents into chunks - the fundamental units that will be embedded and retrieved. Chunking is the most critical step in RAG that determines retrieval quality.
Why Chunking Matters: A Deep Dive
Embeddings work best when text is focused on a single topic or concept. A 100-page 10-K filing contains dozens of different topics. If you embed the entire document as one chunk:
Chunking solves this by creating focused, semantically coherent units.
- โข Irrelevant context dilutes answer
- โข Lower similarity scores
- โข May exceed context limits
- โข Expensive token usage
- โข Focused semantic meaning
- โข Strong similarity scores
- โข Self-contained context
- โข Efficient retrieval
- โข Missing surrounding context
- โข Fragments tables/lists
- โข Need more chunks = slower
- โข Incomplete information
Chunking Strategies
1. Fixed-Size Chunking
SimpleSplit text every N characters or tokens. Add overlap to preserve context across boundaries.
from langchain.text_splitter import CharacterTextSplitter
splitter = CharacterTextSplitter(
chunk_size=500, # ~125 words
chunk_overlap=50, # 10% overlap
separator="\n"
)
chunks = splitter.split_text(document)
print(f"Created {len(chunks)} chunks")- โข Simple to implement
- โข Predictable chunk sizes
- โข Fast processing
- โข May break mid-sentence
- โข Ignores document structure
- โข Not semantically aware
2. Recursive/Sentence-Based Chunking
RecommendedSplit on multiple separators hierarchically: paragraphs first, then sentences, then words. Preserves natural boundaries.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
separators=[
"\n\n", # Double newline (paragraphs)
"\n", # Single newline
". ", # Sentences
" ", # Words
"" # Characters (last resort)
]
)
chunks = splitter.split_text(document)- โข Respects sentence boundaries
- โข More readable chunks
- โข Better semantic coherence
- โข MD&A sections
- โข Letters to shareholders
- โข Narrative text
3. Semantic Chunking
AdvancedSplit when topic changes by analyzing embedding similarity between sentences. Groups semantically related content together.
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai.embeddings import OpenAIEmbeddings
splitter = SemanticChunker(
OpenAIEmbeddings(),
breakpoint_threshold_type="percentile", # or "standard_deviation"
breakpoint_threshold_amount=75
)
chunks = splitter.create_documents([document])- โข Topic-aware splitting
- โข Natural semantic boundaries
- โข Best coherence
- โข Expensive (embedding calls)
- โข Slower processing
- โข Variable chunk sizes
4. Structure-Aware Chunking
Finance-SpecificRespect document structure: sections, headers, tables, lists. Critical for structured financial documents.
- 1.Keep tables intact: Don't split rows/columns. Convert to markdown if needed.
- 2.Preserve lists: Keep bulleted/numbered lists together as one chunk.
- 3.Split by headers: Use H1, H2, H3 tags as natural boundaries.
- 4.Include context: Add section header to each chunk (e.g., "Item 1A - Risk Factors")
5. Parent-Child Chunking
HybridCreate small chunks for precise matching but retrieve larger parent chunks for full context. Best of both worlds.
- 1.Create small "child" chunks (128-256 tokens) for embedding/search
- 2.Link each child to a larger "parent" chunk (512-1024 tokens)
- 3.During retrieval: search using child chunks, return parent chunks
- 4.LLM receives larger context without losing search precision
Understanding Metadata: The Secret Weapon
Metadata is structured information about your chunks that enables powerful filtering, attribution, and context enhancement. Every chunk should have rich metadata attached.
What is Metadata?
Metadata is data about data. For a chunk from a 10-K filing, metadata might include: source file name, page number, section name, company name, fiscal year, filing date, chunk ID, etc.
Analogy: Think of chunks as books in a library. Metadata is the card catalog information - author, publication date, genre, shelf location. It helps you find and contextualize the content.
Why Metadata is Critical
"Find risks from 2024 filings only" โ Filter by year before embedding search
Enable citations like "According to Apple 10-K 2024, page 45..."
Add document hierarchy: "Item 1A โ Risk Factors โ Credit Risk"
In production, isolate chunks by user, organization, or tenant
Example: Well-Structured Metadata
{
"chunk_id": "apple_10k_2024_chunk_127",
"source_file": "apple_10k_2024.pdf",
"source_type": "10-K",
"company": "Apple Inc.",
"ticker": "AAPL",
"fiscal_year": 2024,
"filing_date": "2024-11-01",
"page_number": 45,
"section": "Item 1A",
"section_title": "Risk Factors",
"subsection": "Credit Risk",
"section_path": "Item 1A > Risk Factors > Credit Risk",
"chunk_index": 127,
"total_chunks": 542,
"char_count": 1847,
"token_count": 461,
"created_at": "2024-11-15T10:23:45Z",
"embedding_model": "text-embedding-3-small",
"language": "en",
"tags": ["risk", "credit", "lending"]
}Adding Metadata During Chunking
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.docstore.document import Document
# After extracting text from document
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_text(document_text)
# Add rich metadata to each chunk
documents = []
for i, chunk_text in enumerate(chunks):
metadata = {
"source": "apple_10k_2024.pdf",
"company": "Apple Inc.",
"year": 2024,
"section": current_section, # Track from document structure
"page": current_page,
"chunk_id": i,
"total_chunks": len(chunks)
}
doc = Document(
page_content=chunk_text,
metadata=metadata
)
documents.append(doc)
# Now when you embed and store, metadata goes with each chunk
embeddings = embed_model.embed_documents([doc.page_content for doc in documents])
# Store in vector DB with metadata
for doc, embedding in zip(documents, embeddings):
vector_db.upsert({
"id": doc.metadata["chunk_id"],
"values": embedding,
"metadata": doc.metadata # Stored alongside vector!
})Querying with Metadata Filters
# Example: Pinecone query with metadata filtering
query_embedding = embed_model.embed_query("What are the key credit risks?")
results = index.query(
vector=query_embedding,
top_k=5,
filter={
"year": {"$eq": 2024}, # Only 2024 filings
"section": {"$in": ["Item 1A", "Item 7"]}, # Only risk/MD&A sections
"company": {"$eq": "Apple Inc."} # Only Apple docs
},
include_metadata=True
)
# Retrieved chunks come with full metadata for attribution
for match in results.matches:
print(f"Source: {match.metadata['source']}, Page: {match.metadata['page']}")
print(f"Section: {match.metadata['section_path']}")
print(f"Content: {match.metadata['text'][:200]}...")
print(f"Relevance: {match.score}")
print("---")Chunking Best Practices
Step 4: Generating Embeddings & Vector Storage
Now that you have well-structured chunks with metadata, it's time to convert them into embeddings (dense vector representations) and store them in a vector database for fast semantic search.
Why Embeddings?
Traditional keyword search (like Ctrl+F) only finds exact matches. If you search for "credit risk" you won't find "default probability" even though they're semantically related. Embeddings solve this by capturing semantic meaning.
Query: "What are the main risks?"
Misses: "challenges", "threats", "vulnerabilities", "exposures" - all semantically similar but different words
Query: "What are the main risks?"
Finds: risks, challenges, threats, vulnerabilities, exposures - all semantically similar concepts
Embeddings are vectors (arrays of numbers) that encode semantic meaning. Similar concepts have similar vectors (measured by cosine similarity, dot product, etc.).
Choosing an Embedding Model
Different embedding models have different strengths. Key factors: dimensions, performance, cost, and domain specificity.
OpenAI text-embedding-3-small
RecommendedOpenAI text-embedding-3-large
PremiumCohere embed-english-v3.0
AlternativeSentence Transformers (Open Source)
Self-HostedGenerating Embeddings: Implementation
Simple Example: Single Chunk
from openai import OpenAI
client = OpenAI(api_key="your-api-key")
# Your chunk text
chunk_text = "The company faces significant credit risk in its commercial lending portfolio..."
# Generate embedding
response = client.embeddings.create(
model="text-embedding-3-small",
input=chunk_text
)
embedding = response.data[0].embedding # 1,536-dimensional vector
print(f"Embedding dimensions: {len(embedding)}")
# Output: Embedding dimensions: 1536Production: Batch Processing
For large document sets, process chunks in batches to optimize API calls and handle rate limits.
from openai import OpenAI
from tqdm import tqdm
client = OpenAI(api_key="your-api-key")
# Your chunks from previous step (list of Document objects with metadata)
chunks = [...] # List of chunk texts
# Batch size (OpenAI allows up to 2048 inputs per request)
BATCH_SIZE = 100
embeddings = []
for i in tqdm(range(0, len(chunks), BATCH_SIZE)):
batch = chunks[i:i + BATCH_SIZE]
response = client.embeddings.create(
model="text-embedding-3-small",
input=[chunk.page_content for chunk in batch]
)
# Extract embeddings from response
batch_embeddings = [data.embedding for data in response.data]
embeddings.extend(batch_embeddings)
print(f"Generated {len(embeddings)} embeddings")
# Now embeddings[i] corresponds to chunks[i]Storing in Vector Database
Once you have embeddings, store them in a vector database alongside metadata for fast similarity search.
Example: Pinecone (Managed Vector DB)
from pinecone import Pinecone, ServerlessSpec
# Initialize Pinecone
pc = Pinecone(api_key="your-pinecone-key")
# Create index (one-time setup)
index_name = "financial-docs"
if index_name not in pc.list_indexes().names():
pc.create_index(
name=index_name,
dimension=1536, # Must match embedding model dimensions
metric="cosine", # or "dotproduct", "euclidean"
spec=ServerlessSpec(cloud="aws", region="us-east-1")
)
index = pc.Index(index_name)
# Prepare data for upsert (chunk + embedding + metadata)
vectors_to_upsert = []
for i, (chunk, embedding) in enumerate(zip(chunks, embeddings)):
vectors_to_upsert.append({
"id": f"chunk_{i}", # Unique ID
"values": embedding, # The 1536-dim vector
"metadata": {
# All your rich metadata from chunking step!
"text": chunk.page_content, # Store original text too
"source": chunk.metadata["source"],
"company": chunk.metadata["company"],
"year": chunk.metadata["year"],
"section": chunk.metadata["section"],
"page": chunk.metadata["page"]
}
})
# Upsert in batches (Pinecone recommends 100-200 per batch)
UPSERT_BATCH_SIZE = 100
for i in range(0, len(vectors_to_upsert), UPSERT_BATCH_SIZE):
batch = vectors_to_upsert[i:i + UPSERT_BATCH_SIZE]
index.upsert(vectors=batch)
print(f"Uploaded {len(vectors_to_upsert)} vectors to Pinecone")
# Check index stats
stats = index.describe_index_stats()
print(f"Total vectors in index: {stats.total_vector_count}")Example: Chroma (Local/Open Source)
import chromadb
from chromadb.config import Settings
# Initialize Chroma (local storage)
client = chromadb.Client(Settings(
persist_directory="./chroma_db"
))
# Create or get collection
collection = client.get_or_create_collection(
name="financial_docs",
metadata={"description": "Financial document embeddings"}
)
# Add documents with embeddings and metadata
collection.add(
ids=[f"chunk_{i}" for i in range(len(chunks))],
embeddings=embeddings,
documents=[chunk.page_content for chunk in chunks],
metadatas=[chunk.metadata for chunk in chunks]
)
print(f"Collection size: {collection.count()}")
# Chroma handles embedding generation too! (simpler but less control)
# collection.add(
# ids=[...],
# documents=[...], # Just text, Chroma embeds automatically
# metadatas=[...]
# )Index Optimization for Production
1. Namespaces (Multi-Tenancy)
Use namespaces to isolate data by customer, department, or use case in a single index.
2. Metadata Indexes
Index frequently-filtered metadata fields for faster filtered queries (year, company, section).
3. Dimension Reduction
OpenAI allows reducing dimensions (e.g., 1536 โ 512) to save storage with minimal quality loss.
4. Incremental Updates
When docs change, upsert with same ID to update vectors. Use timestamps in metadata to track freshness.
Embeddings & Storage Best Practices
Step 5: Query Phase & Advanced Retrieval
The indexing phase is done (extract โ chunk โ embed โ store). Now a user asks a question. The query phase retrieves the most relevant chunks to answer that question.
Basic Query Flow
Similarity Search Methods
How do we measure "similarity" between query embedding and chunk embeddings? Three common metrics:
1. Cosine Similarity (Most Common)
RecommendedMeasures the angle between two vectors. Range: -1 (opposite) to 1 (identical). Ignores magnitude, only cares about direction.
โ Works well even if vectors have different magnitudes
2. Dot Product
FastMultiplies corresponding elements and sums. Higher = more similar. Faster than cosine but sensitive to vector magnitude.
โ Slightly faster computation
3. Euclidean Distance (L2)
AlternativeStraight-line distance between two points. Lower = more similar (opposite of above). Less common for embeddings.
โ Less common for modern embedding models
Basic Query Implementation
from openai import OpenAI
from pinecone import Pinecone
# Initialize
client = OpenAI(api_key="your-openai-key")
pc = Pinecone(api_key="your-pinecone-key")
index = pc.Index("financial-docs")
# User query
user_query = "What were Apple's main credit risks in 2024?"
# Step 1: Embed query (SAME model as indexing!)
response = client.embeddings.create(
model="text-embedding-3-small", # Must match indexing model
input=user_query
)
query_embedding = response.data[0].embedding
# Step 2: Search vector DB
results = index.query(
vector=query_embedding,
top_k=5, # Retrieve top 5 most similar chunks
include_metadata=True # Get text + metadata
)
# Step 3: Extract retrieved chunks
retrieved_chunks = []
for match in results.matches:
retrieved_chunks.append({
"text": match.metadata["text"],
"source": match.metadata["source"],
"page": match.metadata["page"],
"score": match.score # Similarity score (0-1 for cosine)
})
# Print results
for i, chunk in enumerate(retrieved_chunks):
print(f"\n--- Result {i+1} (Score: {chunk['score']:.4f}) ---")
print(f"Source: {chunk['source']}, Page: {chunk['page']}")
print(f"Text: {chunk['text'][:200]}...")
# Next: Pass retrieved_chunks to LLM (covered in next step)Advanced Retrieval Techniques
Basic vector search works, but production RAG systems use these advanced techniques to improve retrieval quality:
1. Hybrid Search (Vector + Keyword)
ProductionCombine semantic search (embeddings) with keyword search (BM25). Best for queries with specific terms like "EBITDA", "10-K", ticker symbols.
- โข Vector search: "What are credit risks?" โ finds "default probability", "loan exposure"
- โข Keyword search: "Find AAPL mentions" โ exact match for ticker symbol
- โข Hybrid: Combines both strengths
# Weaviate supports hybrid search natively
results = collection.query.hybrid(
query="What is EBITDA for AAPL in 2024?",
alpha=0.75, # 0=pure keyword, 1=pure vector, 0.75=mostly vector
limit=5
)
# Or combine results manually
vector_results = index.query(vector=query_emb, top_k=10)
keyword_results = bm25_search(query_text, top_k=10)
combined = rerank_and_merge(vector_results, keyword_results)2. Reranking
High ImpactRetrieve top-k candidates (e.g., 20), then use a more powerful cross-encoder model to rerank them. Improves accuracy significantly.
- โข Stage 1: Fast vector search โ get 20 candidates
- โข Stage 2: Slow but accurate reranker โ pick best 5
- โข Rerankers: Cohere Rerank, cross-encoders (sentence-transformers)
import cohere
co = cohere.Client("your-cohere-key")
# Step 1: Get initial candidates
results = index.query(vector=query_emb, top_k=20)
candidates = [match.metadata["text"] for match in results.matches]
# Step 2: Rerank with Cohere
reranked = co.rerank(
query=user_query,
documents=candidates,
top_n=5, # Keep only top 5 after reranking
model="rerank-english-v3.0"
)
# Use reranked results
for result in reranked.results:
print(f"Index: {result.index}, Score: {result.relevance_score}")
print(f"Text: {candidates[result.index][:200]}...")3. Query Decomposition
Complex QueriesBreak complex queries into simpler sub-queries, retrieve for each, then combine results.
- 1. "Apple revenue growth 2024"
- 2. "Microsoft revenue growth 2024"
- 3. "Apple profitability 2024"
- 4. "Microsoft profitability 2024"
4. Query Expansion
Recall BoostGenerate multiple variations of the query to improve recall. Use LLM to rephrase or add synonyms.
- โข "What are the main risks?"
- โข "Key risk factors"
- โข "Primary threats and vulnerabilities"
- โข "Critical business challenges"
5. Metadata Filtering
EssentialPre-filter chunks by metadata BEFORE vector search. Dramatically improves relevance and reduces search space.
- โข "Find risks in 2024 filings only" โ filter by year=2024
- โข "Search Apple docs" โ filter by company="Apple Inc."
- โข "MD&A section only" โ filter by section="Item 7"
results = index.query(
vector=query_embedding,
top_k=5,
filter={
"year": {"$eq": 2024},
"company": {"$in": ["Apple Inc.", "Microsoft Corp."]},
"section": {"$eq": "Item 1A"} # Risk Factors only
},
include_metadata=True
)6. Parent Document Retrieval
Context-RichSearch with small child chunks (precise matching), but retrieve larger parent chunks (more context for LLM).
- โข Index: Small chunks (128 tokens) with parent_id in metadata
- โข Search: Find most similar small chunks
- โข Retrieve: Return larger parent chunks (512 tokens)
- โข Result: Precise search + rich context for answering
# Search child chunks child_results = index.query(vector=query_emb, top_k=5) # Get parent IDs parent_ids = [match.metadata["parent_id"] for match in child_results.matches] # Retrieve full parent chunks (from separate store or DB) parent_chunks = fetch_parent_chunks(parent_ids) # Pass parent chunks to LLM (more context!)
Query Phase Best Practices
Step 6: LLM Generation & Response Presentation
You've retrieved the most relevant chunks. Now construct a prompt with this context and send it to the LLM to generate a grounded, accurate answer with proper citations.
Constructing the RAG Prompt
A RAG prompt has 3 essential components: System Prompt, Retrieved Context, and User Question.
1. System Prompt (Instructions)
Tell the LLM its role, constraints, and how to handle the context. Critical: instruct it to ONLY use provided context and cite sources.
2. Retrieved Context (From Vector DB)
The chunks retrieved in the previous step, formatted with source information for attribution.
3. User Question
The original query the user asked.
Production RAG Prompt Template
# System Prompt
system_prompt = """
You are a financial analyst assistant helping users understand financial documents.
CRITICAL INSTRUCTIONS:
1. Answer questions using ONLY the information provided in the Context below
2. If the answer is not in the Context, say "I don't have enough information to answer that based on the provided documents."
3. ALWAYS cite your sources using [Source X] notation
4. Be precise and factual - do not make up information
5. If the Context contains conflicting information, acknowledge it
6. For numerical data, cite the exact source and date
Context will include the source document, page number, and section for each chunk.
"""
# Format retrieved chunks with metadata
context_parts = []
for i, chunk in enumerate(retrieved_chunks):
source_info = f"[Source {i+1}] {chunk['source']}, Page {chunk['page']}, Section: {chunk['section']}"
context_parts.append(f"{source_info}\n{chunk['text']}\n")
context = "\n".join(context_parts)
# Send to LLM
response = client.chat.completions.create(
model="gpt-4-turbo",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"CONTEXT:\n{context}\n\nQUESTION: {user_query}"}
],
temperature=0.1, # Low temperature for factual answers
max_tokens=1000
)
answer = response.choices[0].message.contentCitation & Attribution Strategies
Citations are CRITICAL for trust, especially in finance. Multiple approaches:
Approach 1: Inline Citations (Recommended)
Instruct LLM to cite after each claim using [Source X] notation.
Approach 2: Structured JSON Output
Request structured output with separate answer and citations. Better for programmatic use.
{
"answer": "Apple's primary credit risks...",
"citations": [
{
"claim": "exposure to commercial lending portfolios",
"source": "Apple 10-K 2024",
"page": 45,
"section": "Item 1A - Risk Factors",
"confidence": "high"
}
],
"has_sufficient_context": true
}Approach 3: Post-Processing Citation Extraction
Generate answer, then use a second LLM call to extract and verify citations from retrieved chunks.
โ Can verify each claim is actually supported by context
Response Quality Controls
1. Hallucination Prevention
2. Handling Insufficient Context
# Add to system prompt:
"If the provided context does not contain enough
information to fully answer, say:
'Based on the provided documents, I can only
partially answer: [what you found]. However,
I don't have information about [what's missing].'"
# Or use confidence scoring:
{
"answer": "...",
"confidence": "partial",
"missing_info": ["metric not found"]
}3. Conflicting Information
- โข Acknowledge the conflict explicitly
- โข Cite both sources with dates
- โข Clarify which is more recent
Streaming Responses (Better UX)
For production systems, stream the LLM response token-by-token for better perceived performance.
stream = client.chat.completions.create(
model="gpt-4-turbo",
messages=[...],
stream=True
)
for chunk in stream:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")LLM Generation Best Practices
Advanced RAG Techniques
Building RAG on Financial Documents
Let's build a production-ready RAG system for financial documents. We'll use Pinecone as our vector database and OpenAI embeddings, but the principles apply to any stack.
Complete End-to-End RAG Implementation
This production-ready implementation demonstrates all 7 steps of the RAG pipeline: document loading, tokenization, chunking with rich metadata, embeddings, vector storage, advanced querying, and LLM generation with citations.
Step 1: Setup and Dependencies
# Install dependencies pip install pinecone-client openai pdfplumber tiktoken python-dotenv tqdm # .env file OPENAI_API_KEY=your_openai_key PINECONE_API_KEY=your_pinecone_key PINECONE_ENVIRONMENT=us-east-1 # Imports import pdfplumber import tiktoken from openai import OpenAI from pinecone import Pinecone, ServerlessSpec from tqdm import tqdm import os from datetime import datetime
Step 2: Document Loading & Extraction
Using PDFPlumber for better table extraction from financial documents
# Load 10-K filing with PDFPlumber (better for financial docs)
def extract_document_with_metadata(pdf_path: str):
"""Extract text and metadata from financial PDF"""
pages_data = []
with pdfplumber.open(pdf_path) as pdf:
for page_num, page in enumerate(pdf.pages, 1):
# Extract text
text = page.extract_text()
# Extract tables (important for financial docs!)
tables = page.extract_tables()
# Combine text and tables
page_content = text
if tables:
for table in tables:
# Convert table to readable text
table_text = "\n".join([" | ".join(row) for row in table])
page_content += f"\n\nTABLE:\n{table_text}"
pages_data.append({
"page_num": page_num,
"content": page_content,
"has_tables": len(tables) > 0
})
return pages_data
# Extract document
pdf_path = "apple_10k_2024.pdf"
pages = extract_document_with_metadata(pdf_path)
print(f"Extracted {len(pages)} pages from {pdf_path}")Step 3: Tokenization & Chunking with Rich Metadata
Chunk documents while preserving metadata for filtering and attribution
# Initialize tokenizer for token counting
encoding = tiktoken.encoding_for_model("gpt-4")
def count_tokens(text: str) -> int:
"""Count tokens in text"""
return len(encoding.encode(text))
def chunk_document_with_metadata(pages, chunk_size=500, overlap=50):
"""
Chunk document with comprehensive metadata
chunk_size and overlap are in tokens
"""
chunks = []
for page in pages:
text = page["content"]
page_num = page["page_num"]
# Simple sentence-based chunking
sentences = text.split(". ")
current_chunk = ""
current_tokens = 0
for sentence in sentences:
sentence_tokens = count_tokens(sentence)
# If adding this sentence exceeds chunk size, save current chunk
if current_tokens + sentence_tokens > chunk_size and current_chunk:
chunks.append({
"text": current_chunk.strip(),
"metadata": {
# Identification
"chunk_id": f"chunk_{len(chunks)}",
"source_file": "apple_10k_2024.pdf",
# Document Classification
"document_type": "10-K",
"company": "Apple Inc.",
"ticker": "AAPL",
"fiscal_year": 2024,
"filing_date": "2024-11-01",
# Location Information
"page_number": page_num,
"section": "Item 1A", # You'd parse this from ToC
"section_title": "Risk Factors",
# Technical Metadata
"chunk_tokens": count_tokens(current_chunk),
"has_tables": page["has_tables"],
"created_at": datetime.now().isoformat(),
# Multi-tenancy (if needed)
"tenant_id": "acme_corp",
"user_access_level": "analyst"
}
})
# Start new chunk with overlap
current_chunk = sentence + ". "
current_tokens = sentence_tokens
else:
current_chunk += sentence + ". "
current_tokens += sentence_tokens
# Add remaining chunk
if current_chunk.strip():
chunks.append({
"text": current_chunk.strip(),
"metadata": {
"chunk_id": f"chunk_{len(chunks)}",
"source_file": "apple_10k_2024.pdf",
"document_type": "10-K",
"company": "Apple Inc.",
"ticker": "AAPL",
"fiscal_year": 2024,
"page_number": page_num,
"chunk_tokens": count_tokens(current_chunk),
"has_tables": page["has_tables"]
}
})
return chunks
# Create chunks
chunks = chunk_document_with_metadata(pages, chunk_size=500, overlap=50)
print(f"Created {len(chunks)} chunks")
print(f"Average tokens per chunk: {sum(c['metadata']['chunk_tokens'] for c in chunks) / len(chunks):.0f}")Step 4: Generate Embeddings & Store in Pinecone
Batch embed chunks and store with metadata for efficient retrieval
# Initialize OpenAI and Pinecone
client = OpenAI()
pc = Pinecone(api_key=os.getenv("PINECONE_API_KEY"))
# Create index (serverless for cost efficiency)
index_name = "financial-docs-rag"
if index_name not in pc.list_indexes().names():
pc.create_index(
name=index_name,
dimension=1536, # text-embedding-3-small dimension
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1")
)
index = pc.Index(index_name)
# Batch embed and upload
BATCH_SIZE = 100
def embed_and_upload_chunks(chunks, batch_size=100):
"""Embed chunks in batches and upload to Pinecone"""
for i in tqdm(range(0, len(chunks), batch_size), desc="Embedding & uploading"):
batch = chunks[i:i+batch_size]
# Get embeddings for batch
texts = [chunk["text"] for chunk in batch]
response = client.embeddings.create(
model="text-embedding-3-small",
input=texts
)
# Prepare vectors for Pinecone
vectors = []
for j, embedding_obj in enumerate(response.data):
chunk = batch[j]
vectors.append({
"id": chunk["metadata"]["chunk_id"],
"values": embedding_obj.embedding,
"metadata": {
"text": chunk["text"], # Store full text
**chunk["metadata"] # All metadata
}
})
# Upload batch to Pinecone
index.upsert(vectors=vectors)
print(f"โ Indexed {len(chunks)} chunks with metadata")
# Upload all chunks
embed_and_upload_chunks(chunks, batch_size=BATCH_SIZE)Step 5: Query with Advanced Retrieval & Metadata Filtering
Use metadata filters to narrow search scope and improve relevance
def retrieve_chunks(
query: str,
top_k: int = 5,
filters: dict = None
):
"""
Retrieve relevant chunks with optional metadata filtering
Example filters:
- {"fiscal_year": 2024}
- {"company": "Apple Inc.", "section": "Risk Factors"}
- {"has_tables": True}
"""
# 1. Embed the query
response = client.embeddings.create(
model="text-embedding-3-small",
input=query
)
query_embedding = response.data[0].embedding
# 2. Search Pinecone with metadata filtering
search_params = {
"vector": query_embedding,
"top_k": top_k,
"include_metadata": True
}
# Add metadata filters if provided
if filters:
search_params["filter"] = filters
results = index.query(**search_params)
# 3. Extract and format results
retrieved_chunks = []
for match in results.matches:
retrieved_chunks.append({
"id": match.id,
"text": match.metadata.get("text", ""),
"score": match.score,
"metadata": {
k: v for k, v in match.metadata.items()
if k != "text" # Exclude text from metadata display
}
})
return retrieved_chunks
# Example: Retrieve chunks about risks from 2024 filing
retrieved = retrieve_chunks(
query="What are the main credit risks?",
top_k=5,
filters={"fiscal_year": 2024, "section": "Item 1A"} # Only risk factors
)
print(f"Retrieved {len(retrieved)} chunks")
for i, chunk in enumerate(retrieved, 1):
print(f"\n[{i}] Score: {chunk['score']:.3f}")
print(f"Source: {chunk['metadata']['source_file']}, Page {chunk['metadata']['page_number']}")
print(f"Text preview: {chunk['text'][:200]}...")Step 6: LLM Generation with Citations
Generate answers grounded in retrieved context with proper citations
def generate_answer(query: str, retrieved_chunks: list):
"""Generate answer from retrieved chunks with citations"""
# 1. Build context with source attribution
context_parts = []
for i, chunk in enumerate(retrieved_chunks, 1):
context_parts.append(
f"[Source {i}]\n"
f"Company: {chunk['metadata']['company']}\n"
f"Document: {chunk['metadata']['document_type']} ({chunk['metadata']['fiscal_year']})\n"
f"Section: {chunk['metadata']['section_title']}\n"
f"Page: {chunk['metadata']['page_number']}\n"
f"Content: {chunk['text']}\n"
)
context = "\n---\n".join(context_parts)
# 2. Build RAG prompt with strict instructions
system_prompt = """You are a financial analyst assistant.
CRITICAL RULES:
1. Answer ONLY using information from the provided sources
2. ALWAYS cite sources using [Source N] notation after each claim
3. If information is not in the sources, explicitly state "This information is not available in the provided documents"
4. Be specific with numbers, dates, and facts
5. If sources conflict, mention both viewpoints and cite each"""
user_prompt = f"""Based on the following sources, answer this question:
QUESTION: {query}
SOURCES:
{context}
Provide a comprehensive answer with inline citations."""
# 3. Generate answer (low temperature for factuality)
completion = client.chat.completions.create(
model="gpt-4",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt}
],
temperature=0.1 # Low temperature for factuality
)
answer = completion.choices[0].message.content
# 4. Return structured response
return {
"answer": answer,
"sources": [
{
"id": i,
"company": chunk["metadata"]["company"],
"document": f"{chunk['metadata']['document_type']} {chunk['metadata']['fiscal_year']}",
"section": chunk["metadata"]["section_title"],
"page": chunk["metadata"]["page_number"],
"relevance_score": chunk["score"]
}
for i, chunk in enumerate(retrieved_chunks, 1)
],
"query": query
}
# Example usage in complete RAG pipeline
def query_rag(question: str, filters: dict = None, top_k: int = 5):
"""Complete RAG pipeline: retrieve + generate"""
# Step 1: Retrieve relevant chunks
retrieved = retrieve_chunks(question, top_k=top_k, filters=filters)
# Step 2: Generate answer
result = generate_answer(question, retrieved)
return resultStep 7: Complete Example Usage
# Query the RAG system
result = query_rag(
question="What credit risks did Apple disclose in their 2024 10-K filing?",
filters={"fiscal_year": 2024, "company": "Apple Inc."},
top_k=5
)
# Display results
print("=" * 80)
print("QUESTION:", result["query"])
print("=" * 80)
print("\nANSWER:")
print(result["answer"])
print("\n" + "=" * 80)
print("SOURCES:")
for source in result["sources"]:
print(f" [{source['id']}] {source['document']} - {source['section']}")
print(f" Page {source['page']} (Relevance: {source['relevance_score']:.3f})")
print("=" * 80)
# Example Output:
# ===============================================================================
# QUESTION: What credit risks did Apple disclose in their 2024 10-K filing?
# ===============================================================================
#
# ANSWER:
# According to Apple's 2024 10-K filing, the company disclosed several credit risks:
#
# 1. **Counterparty Credit Risk**: Apple faces credit risk from financial institutions
# holding its cash and investments [Source 1]. The company attempts to mitigate
# this by diversifying across multiple highly-rated institutions.
#
# 2. **Customer Credit Risk**: The company has credit exposure to customers through
# accounts receivable and financing arrangements [Source 2]. This is particularly
# relevant for carrier partners and enterprise customers with payment terms.
#
# 3. **Sovereign Credit Risk**: Apple has exposure to sovereign debt of various
# countries where it operates [Source 3], which could be impacted by deteriorating
# credit conditions in those jurisdictions.
#
# ===============================================================================
# SOURCES:
# [1] 10-K 2024 - Risk Factors
# Page 23 (Relevance: 0.887)
# [2] 10-K 2024 - Risk Factors
# Page 24 (Relevance: 0.856)
# [3] 10-K 2024 - Risk Factors
# Page 25 (Relevance: 0.823)
# ===============================================================================Key Implementation Notes
Step 7: Evaluation & Metrics
How do you know if your RAG system is working well? You need systematic evaluation across retrieval quality, answer quality, latency, and cost.
1. Retrieval Quality Metrics
Measure how well the vector search retrieves relevant chunks. Requires a labeled test set with queries and known relevant documents.
Precision@k
Of the top k retrieved chunks, what percentage are actually relevant?
Recall@k
Of all relevant chunks in the corpus, what percentage did we retrieve in top k?
Mean Reciprocal Rank (MRR)
Measures how high the first relevant result appears. Rewards having the best result at the top.
Normalized Discounted Cumulative Gain (NDCG@k)
Most sophisticated metric. Considers both relevance and ranking position. Higher-ranked relevant docs contribute more.
Computing Retrieval Metrics
# Test dataset: queries with labeled relevant chunk IDs
test_queries = [
{
"query": "What are Apple's credit risks?",
"relevant_chunk_ids": ["chunk_45", "chunk_127", "chunk_203"]
},
# ... more test cases
]
# Evaluate retrieval
def evaluate_retrieval(test_queries, retrieval_function, k=5):
precisions, recalls, mrrs, ndcgs = [], [], [], []
for test_case in test_queries:
# Retrieve top k chunks
retrieved = retrieval_function(test_case["query"], k=k)
retrieved_ids = [chunk["id"] for chunk in retrieved]
relevant_ids = set(test_case["relevant_chunk_ids"])
# Precision@k
relevant_in_k = len(set(retrieved_ids) & relevant_ids)
precision = relevant_in_k / k
precisions.append(precision)
# Recall@k
recall = relevant_in_k / len(relevant_ids) if relevant_ids else 0
recalls.append(recall)
# MRR
for rank, chunk_id in enumerate(retrieved_ids, 1):
if chunk_id in relevant_ids:
mrrs.append(1.0 / rank)
break
else:
mrrs.append(0)
# NDCG (simplified - use sklearn for full implementation)
# ... NDCG calculation
return {
"precision@k": np.mean(precisions),
"recall@k": np.mean(recalls),
"MRR": np.mean(mrrs),
"NDCG@k": np.mean(ndcgs)
}
results = evaluate_retrieval(test_queries, my_rag_retrieval, k=5)
print(results)
# {'precision@5': 0.74, 'recall@5': 0.62, 'MRR': 0.81, 'NDCG@5': 0.79}2. Answer Quality Metrics
Retrieval is only half the story. The final answer quality matters most.
LLM-as-Judge (Automated)
Use a strong LLM (GPT-4) to evaluate answer quality on multiple dimensions.
judge_prompt = """
Rate the answer on a scale of 1-5 for each dimension:
QUESTION: {query}
RETRIEVED CONTEXT: {context}
GENERATED ANSWER: {answer}
Evaluate:
1. Factual Accuracy: Is every claim supported by context?
2. Completeness: Does it fully answer the question?
3. Citation Quality: Are sources cited properly?
4. Relevance: Is the answer on-topic?
Return JSON:
{
"factual_accuracy": 1-5,
"completeness": 1-5,
"citation_quality": 1-5,
"relevance": 1-5,
"overall": 1-5,
"reasoning": "brief explanation"
}
"""
evaluation = judge_llm.generate(judge_prompt)
scores = json.loads(evaluation)Human Evaluation (Gold Standard)
Human experts rate answers. Most accurate but expensive and slow.
- โข Sample 100-500 diverse queries
- โข Have 2-3 domain experts rate each answer
- โข Use rubric: factuality, completeness, citation, clarity
- โข Calculate inter-rater agreement (Cohen's kappa)
Groundedness/Faithfulness
Measure if the answer is grounded in retrieved context (doesn't hallucinate).
User Feedback (Production Signal)
Thumbs up/down, ratings, explicit feedback. Your best real-world signal.
- โข Positive feedback rate
- โข Time to thumbs down (fast = obviously bad)
- โข Follow-up questions (indicates insufficient answer)
- โข Query reformulations (indicates misunderstanding)
3. System Performance Metrics
Latency Tracking
- โข Query embedding time: ~50-100ms
- โข Vector search time: ~50-200ms
- โข LLM generation time: 1-5s
- โข End-to-end latency: <3s target
Cost Monitoring
- โข Embedding API calls ($/query)
- โข LLM tokens (input + output)
- โข Vector DB costs (storage + queries)
- โข Reranking API costs (if used)
Error Rates
Cache Hit Rates
Evaluation Best Practices
Production Best Practices
Taking RAG from prototype to production requires attention to reliability, security, compliance, and observability. Here's what you need to know.
Security & Compliance
- โขAccess Control: Implement document-level permissions. Users should only retrieve documents they're authorized to see.
- โขData Residency: For regulated financial data, consider self-hosted vector DB or region-specific deployments.
- โขPII Handling: Scan and mask sensitive information before embedding. Never log raw queries containing PII.
- โขAudit Logging: Log all queries, retrievals, and responses for compliance and debugging.
Monitoring & Observability
- โขTrack Key Metrics: Latency (p50, p95, p99), error rates, cost per query, retrieval accuracy.
- โขUser Feedback: Implement thumbs up/down for answers. This is your ground truth for improving retrieval.
- โขQuery Analysis: Monitor most common queries, failed queries, and retrieval gaps to improve your system.
- โขData Freshness: Track when documents were last indexed. Alert on stale data in production.
Error Handling & Fallbacks
- โขGraceful Degradation: If vector DB is down, fall back to keyword search or cached responses.
- โขRetry Logic: Implement exponential backoff for API failures (embedding, LLM, vector DB).
- โขNo Results Handling: When retrieval returns no good matches, inform the user rather than hallucinating.
- โขRate Limiting: Protect against abuse with per-user query limits and cost quotas.
Performance Optimization
- โขEmbedding Caching: Cache embeddings for common queries to reduce API calls and latency.
- โขBatch Processing: Index documents in batches rather than one-by-one for 10x speedup.
- โขAsync Operations: Use async/await for parallel embedding and LLM calls to improve throughput.
- โขSmart Chunking: Pre-compute and cache chunk embeddings. Only re-index when documents change.
Production Readiness Checklist
Infrastructure
Security
Quality
Compliance
Common Pitfalls to Avoid
Latest Advancements in Embeddings & RAG
RAG has evolved from a simple retrieve-and-generate pattern into a sophisticated, agent-driven knowledge architecture. Here is what has changed.
GraphRAG โ Beyond Chunk Retrieval
Traditional RAG retrieves the most semantically similar chunks. GraphRAG first builds a knowledge graph from your documents (entities, relationships, communities), then uses that graph to answer questions โ especially multi-hop questions that span many documents.
- โข "What is the exposure of Firm A to all entities that have defaulted in the last 3 quarters?"
- โข "Summarise all the risks mentioned across all 500 client reports"
- โข Questions requiring synthesis across many documents
- โข Multi-hop reasoning across documents
- โข Global summaries over large corpora
- โข Relationship queries (who is connected to whom)
- โข Community detection (thematic clusters)
Finance use: regulatory knowledge graphs (which rule connects to which entity), counterparty risk networks, M&A target relationship mapping, compliance obligation graphs across jurisdictions.
ColPali โ Multimodal RAG on Visual Documents
Traditional RAG on PDFs: convert to text (OCR), chunk the text, embed the text. ColPali embeds the visual page directly as an image โ no OCR, no text extraction. Complex tables, charts, and layouts are preserved perfectly.
PDF โ OCR โ text chunks โ text embeddings โ search
PDF โ page images โ vision-language embeddings โ search
Earnings slides, regulatory tables, fund factsheets โ no data lost in OCR errors
Agentic RAG
Instead of one fixed retrieval step, an agent decides the retrieval strategy dynamically โ and can retrieve multiple times.
Long Context vs RAG
With 1M token windows, sometimes you don't need RAG. When to use which:
Hybrid Search + Reranking โ Now Standard
Hybrid Search
Sparse (BM25/keyword) + Dense (vector) retrieval always outperforms either alone. Most vector databases now support this natively โ Pinecone, Weaviate, Azure AI Search, pgvector all have hybrid search APIs. Use Reciprocal Rank Fusion (RRF) to merge results.
Reranking as Standard
Retrieve 20 candidates, rerank to top 5 with a cross-encoder (Cohere Rerank 3, BGE Reranker). Dramatically improves precision with minimal latency overhead (~100ms). Cohere Rerank 3 supports 100+ languages.