Introduction to Language Models
The Challenge: Teaching Machines to Understand Language
Human language is incredibly complex. Unlike images or structured data, language has sequential dependencies, context, ambiguity, and nuance. How do we teach machines to understand that "bank" means different things in "river bank" versus "savings bank"? Or that "I didn't say she stole my money" has seven different meanings depending on which word you emphasize?
Key Challenges in Natural Language Processing
๐Sequential Dependencies
Words depend on context from earlier in the sentence. "She couldn't fit the trophy in the suitcase because it was too big" - what is "it"?
๐ญAmbiguity & Context
Same words mean different things in different contexts. "Apple" could be a fruit, a company, or even a record label.
๐Variable Length
Sentences can be 3 words or 300 words. Models need to handle any length gracefully.
๐Long-Range Dependencies
Critical information might be separated by many words or sentences. Models must "remember" context across long distances.
Early Approaches: Sequence-to-Sequence Models
Before transformers, the dominant approach was Recurrent Neural Networks (RNNs) and their sequence-to-sequence (seq2seq) architecture. Let's understand why they were revolutionary - and why they ultimately fell short.
Recurrent Neural Networks (RNNs)
Each word is processed sequentially, passing hidden state to the next step
โAdvantages
- โขCan handle variable-length sequences
- โขShares parameters across time steps
- โขMaintains a "memory" via hidden state
โCritical Problems
- โขSequential bottleneck: Must process one word at a time (slow!)
- โขVanishing gradients: Hard to learn long-range dependencies
- โขInformation bottleneck: All context compressed into fixed-size hidden state
LSTM: Long Short-Term Memory Networks
LSTMs were designed to solve the vanishing gradient problem by introducing gates that control information flow
LSTM Cell Architecture
Forget Gate
Decides what information to discard from cell state
Input Gate
Decides what new information to store in cell state
Output Gate
Decides what to output based on cell state
Cell State Update
(Forget some old info + Add some new info = Updated memory)
Improvements over RNN
- โBetter gradient flow via cell state "highway"
- โCan learn longer dependencies (100+ tokens)
- โMore stable training
Still Limitations
- โ Still sequential - cannot parallelize
- โ Struggles with very long sequences (1000+ tokens)
- โ Computationally expensive (4ร gates)
Sequence-to-Sequence with Attention
A breakthrough came with the attention mechanism (Bahdanau et al., 2014) - allowing the model to "focus" on relevant parts of the input
Example: Machine Translation
Encoder (Source Language)
Decoder (Target Language)
Key Insight: When generating "noir" (black), the decoder can attend to the word "black" in the source, even though it appears in a different position!
The Birth of Attention
Attention mechanisms allowed models to create dynamic connections between input and output positions. This was revolutionary, but there was still a problem: the encoder and decoder were still RNNs/LSTMs, meaning they were still sequential and slow.
The question: What if we could use attention EVERYWHERE and get rid of recurrence entirely?
What are Large Language Models (LLMs)?
Large Language Models represent the culmination of decades of NLP research. They are neural networks trained on massive amounts of text data to understand and generate human language. The "large" refers to both the model size (billions of parameters) and training data (trillions of tokens).
LLM Components
Massive Data
Books, websites, code repositories, articles
Transformer Architecture
Attention-based architecture that processes sequences in parallel
Training Process
Learning patterns through next-token prediction
Key Characteristics of LLMs
Unprecedented Scale
Models range from millions to hundreds of billions of parameters. GPT-4 is estimated to have ~1.7 trillion parameters across multiple experts.
Emergent Capabilities
Can perform tasks they weren't explicitly trained for. At certain scales, models suddenly gain abilities like arithmetic, translation, and reasoning.
Contextual Understanding
Understand nuance, context, ambiguity, and can maintain coherent conversations across thousands of tokens.
Transfer Learning
Pre-trained knowledge can be fine-tuned for specific domains with minimal additional data, making them highly adaptable.
๐ผImpact on Financial Services
Customer Service
24/7 intelligent chatbots handling complex queries, understanding context across multiple interactions, and providing personalized financial advice.
Risk Analysis
Processing vast amounts of financial documents, earnings calls, news articles in seconds to identify risks and opportunities.
Compliance & Regulation
Automated monitoring of regulations, policy changes, and ensuring all communications meet compliance requirements.
Research & Analysis
Extracting insights from market data, financial news, analyst reports, and generating comprehensive investment research.
Transformer Architecture
The Transformer, introduced in the landmark 2017 paper "Attention Is All You Need" by Vaswani et al., revolutionized natural language processing. It solved the fundamental problems of RNNs/LSTMs by:
Parallel Processing
All tokens processed simultaneously instead of sequentially
Direct Connections
Every word can attend to every other word directly
Scalability
Can scale to billions of parameters efficiently
High-Level Architecture
ENCODER
DECODER
Note: Modern LLMs like GPT use decoder-only architecture (no encoder). BERT uses encoder-only. Original Transformer used both for translation.
Input Preparation: From Text to Vectors
Step-by-Step Transformation
Text split into tokens (can be words, subwords, or characters)
Each token mapped to unique integer from vocabulary (e.g., 50,000 tokens)
Each ID converted to dense vector (e.g., 768 dimensions). Learned during training.
Add positional information so model knows word order (sin/cos functions)
Why Positional Encoding?
Transformers process all tokens in parallel - they have NO inherent sense of order! We need to inject position information.
Formula (Sinusoidal)
Where: pos = position, i = dimension, d = embedding dimension
Properties
- โUnique encoding for each position
- โCan generalize to longer sequences
- โAllows model to learn relative positions
- โDeterministic (not learned parameters)
Self-Attention: The Core Innovation
Self-attention allows each word to look at all other words in the sequence and decide which ones are most relevant to understanding it. It's like giving each word the ability to ask: "Which other words should I pay attention to?"
Example: "The animal didn't cross the street because it was too tired"
What does "it" refer to?
"it" is just the 10th word. No understanding of what it refers to.
"it" attends strongly to "animal" (not "street"), understanding the reference!
Attention scores for "it"
Query, Key, Value (QKV) Mechanism
Self-attention uses three transformations of the input: Query, Key, and Value
Query (Q)
"What am I looking for?"
The question each token asks
Key (K)
"What do I have to offer?"
What each token advertises
Value (V)
"What information do I give?"
The actual content to pass
Deep Dive: What Are X and W?
๐ฅWhat is X?
X is the input matrix containing the embeddings of all tokens in your sequence. After tokenization and embedding lookup (+ positional encoding), each word becomes a vector of numbers.
seq_length: Number of tokens in your input (e.g., 10 tokens)
d_model: Embedding dimension (e.g., 512 or 768)
| dim 0 | dim 1 | dim 2 | ... | dim 767 | |
|---|---|---|---|---|---|
| The | 0.12 | -0.43 | 0.87 | ... | 0.21 |
| cat | 0.34 | 0.21 | -0.56 | ... | -0.12 |
| sat | -0.23 | 0.67 | 0.45 | ... | 0.89 |
โ๏ธWhat are WQ, WK, WV?
These are learned weight matrices that transform the input embeddings into different representations. Think of them as "projection lenses" that look at the same input from three different perspectives.
d_model: Input embedding dimension (768)
d_k: Output dimension for Q and K (typically 64)
d_v: Output dimension for V (typically same as d_k)
๐ขThe Matrix Multiplication: X ยท W
When we compute Q = X ยท WQ, we're performing matrix multiplication that transforms each token's embedding from d_model dimensions to d_k dimensions.
Concrete Numerical Example
For token "cat" (row 2 of X):
Result: Query vector for "cat"
This query vector represents: "What is 'cat' looking for in other tokens?"
Why Do We Need Three Separate Transformations?
Q (Query): Represents what this token is "looking for" or "asking about". It encodes the search criteria.
K (Key): Represents what this token "advertises" or "offers" to other tokens. It's how tokens describe themselves.
V (Value): The actual information/content that will be passed along once we determine relevance.
Key Insight: Separating these three roles allows the model to learn different aspects of relationships. The Query and Key determine which tokens to pay attention to, while the Value determines what information to extract from those tokens. This is more powerful than using the same representation for all three purposes!
Library Analogy
Query: "I'm looking for books about machine learning"
Key: Each book's index card describing its content
Value: The actual book content
You compare your query with all keys, find relevant matches, then retrieve the corresponding values!
The Attention Formula
Compute Similarity: QKT
Dot product between query and all keys. High value = high relevance.
Scale: / โdk
Divide by square root of key dimension to prevent vanishing gradients
Softmax
Convert scores to probabilities (sum to 1). Determines how much to attend to each token.
Weighted Sum: ร V
Multiply attention weights by values. Output is weighted combination of all values.
Multi-Head Attention: Power in Diversity
Instead of performing attention once, transformers use multiple attention heads in parallel. Each head learns to focus on different aspects of the relationships between tokens.
Why Multiple Heads?
Head 1: Syntax
Focuses on grammatical relationships (subject-verb-object)
Head 2: Coreference
Tracks what pronouns refer to ("it" โ "animal")
Head 3: Semantics
Captures meaning and context relationships
Multi-Head Attention Architecture
The Magic of Multi-Head Attention
- โEach head can attend to different positions and capture different types of relationships
- โSmaller dimension per head (64 vs 512) makes computation efficient
- โModel learns richer representations by combining multiple perspectives
Layer Normalization & Residual Connections
Two critical techniques that enable training deep transformer networks: residual connections (skip connections) and layer normalization.
Residual Connections
Why Residual Connections?
- โEnable gradient flow through deep networks
- โAllow model to learn identity function if needed
- โPreserve information from earlier layers
Layer Normalization
Formula
ฮผ: Mean across features
ฯยฒ: Variance across features
ฮณ, ฮฒ: Learned scale and shift parameters
ฮต: Small constant for numerical stability
Purpose
- โStabilizes training by normalizing activations
- โReduces internal covariate shift
- โAllows higher learning rates
- โApplied to each token's embedding independently
"Add & Norm" Pattern in Transformers
Position-wise Feed Forward Network
After attention, each token passes through a simple 2-layer neural network independently. This is applied to each position separately and identically.
FFN Architecture
Key Characteristics
- โขApplied to each position independently
- โขSame weights shared across all positions
- โขIntermediate layer is 4ร larger (2048 vs 512)
Purpose
- โขAdds non-linearity and complexity
- โขTransforms token representations after attention
- โขContains most of the model's parameters
Mixture of Experts (MoE)
Modern large models like GPT-4, Mixtral, and others use Mixture of Experts to scale efficiently. Instead of one feed-forward network, they have multiple "expert" networks and route each token to a subset of experts.
How MoE Works
Advantages
- โSparse activation: Only 2-4 experts active per token
- โEfficient scaling: 100B+ parameters with cost of ~13B
- โSpecialization: Experts learn different domains/tasks
- โFaster inference than dense models of same quality
Challenges
- โ Load balancing: Ensure all experts are used
- โ Training complexity: Need auxiliary loss functions
- โ Memory requirements: All experts must fit in memory
Models Using MoE
Putting It All Together: The Complete Transformer Block
Single Transformer Layer (Decoder Block)
Key Insights
Attention layer: Allows tokens to communicate and share information
FFN layer: Processes each token independently with non-linear transformations
Residual connections: Preserve information and enable deep architectures
Layer norms: Stabilize training and improve convergence
Full Model: Stack 12-96 of these layers, add final layer normalization and linear projection to vocabulary. That's a transformer!
Training vs Inference: How Transformers Actually Work
Now that we understand the transformer architecture, let's explore how transformers actually work in practice - the critical difference between training (learning from data) and inference (generating new text). Understanding this is essential for deploying LLMs in production financial systems.
Masked Self-Attention: Preventing "Cheating"
In decoder models (GPT, Claude, etc.), we use masked self-attention to prevent tokens from "looking ahead" at future tokens. This ensures the model learns to predict the next token using only past context.
The Problem: Without Masking
Imagine we're training the model on: "The stock price increased"
โ Without Masking (Cheating!):
When predicting "price", the model can see "increased" โ It's cheating by looking at future tokens!
With masking, each token can only attend to itself and previous tokens:
โ With Masking (Proper Training):
When predicting "price", the model can only see "The stock" โ Must learn to predict next word from past context only!
Attention Mask Matrix (Causal Mask)
The mask is a lower-triangular matrix. A value of 1 means "can attend", 0 means "cannot attend" (or -โ in practice before softmax).
| The | stock | price | increased | |
|---|---|---|---|---|
| The | 1 | 0 | 0 | 0 |
| stock | 1 | 1 | 0 | 0 |
| price | 1 | 1 | 1 | 0 |
| increased | 1 | 1 | 1 | 1 |
Green (1): Token can attend to this position
Red (0): Token blocked from attending (future position)
Creating the Causal Mask (PyTorch):
import torch
seq_len = 4 # "The stock price increased"
# Create causal mask (lower triangular)
mask = torch.tril(torch.ones(seq_len, seq_len))
# In practice, convert 0s to -inf before softmax
# This ensures attention weights to future tokens = 0
mask = mask.masked_fill(mask == 0, float('-inf'))
mask = mask.masked_fill(mask == 1, 0.0)
# Applied in attention: softmax(QK^T / sqrt(d) + mask)Why This Matters
- โขTraining: Ensures model learns proper sequential prediction
- โขCausality: Respects the order of information (can't use future to predict past)
- โขAutoregressive: Enables token-by-token generation at inference time
Training Phase: Learning from Data
During training, transformers process entire sequences in parallel using a technique called teacher forcing. This is fast and efficient.
How Training Works: Teacher Forcing
Training Example: "The company reported strong earnings"
INPUT (what we give the model):
TARGET (what we want the model to predict at each position):
KEY INSIGHT - Parallel Processing:
The model predicts ALL positions simultaneously in one forward pass! The causal mask ensures position N can only see positions 0 to N-1.
Loss Computation (Cross-Entropy)
For each position, compute cross-entropy loss:
Position 2 (predicting "reported"):
Target: "reported"
Model outputs probabilities over 50k vocab
Loss = -log P("reported" | "The company")
Backward Pass:
2. Backpropagate gradients
3. Update all model parameters
4. Repeat for next batch
โ Training Advantages
- โขParallel: All tokens processed simultaneously
- โขEfficient: Full GPU utilization, fast training
- โขDense signal: Learn from every token position
- โขTeacher forcing: Always has correct previous tokens
Training Scale
- โขBatch size: 1-8 million tokens per batch
- โขData: Trillions of tokens (entire internet!)
- โขTime: Weeks to months on thousands of GPUs
- โขCost: $10M - $100M+ for frontier models
Inference Phase: Autoregressive Generation
During inference (generation), the model produces text one token at a time in a sequential loop. This is called autoregressive generation and is fundamentally different from training.
Autoregressive Generation Loop
User Prompt: "The company's Q3 revenue"
Final Output:
"The company's Q3 revenue increased by 15% year-over-year."
Generation Flow Diagram
Sampling Strategies: How to Choose the Next Token
The model outputs a probability distribution over 50,000+ tokens. How do we pick which one to generate? Different strategies produce different text characteristics.
1. Greedy Decoding
Cons: Repetitive, boring, can get stuck in loops
2. Temperature Sampling
Original: {increased: 0.45, grew: 0.30}
After T=0.1: {increased: 0.85, grew: 0.12} โ More deterministic
Original: {increased: 0.45, grew: 0.30}
After T=1.5: {increased: 0.38, grew: 0.35} โ More uniform
3. Top-k and Top-p (Nucleus) Sampling
Set others to 0 probability.
Sample from these 40.
Dynamic cutoff based on distribution.
โ Inference Challenges
- โขSequential: Cannot parallelize token generation
- โขSlow: Each token requires full forward pass
- โขLatency: 500 tokens at 20 tokens/sec = 25 seconds!
- โขMemory: Must store KV cache for past tokens
- โขCost: More expensive per query than training per token
โ Optimization Techniques
- โขKV Caching: Reuse past key-value computations
- โขQuantization: Use INT8/INT4 instead of FP16
- โขBatching: Process multiple requests together
- โขSpeculative decoding: Predict multiple tokens ahead
- โขStreaming: Show tokens as generated (better UX)
Financial Services: Training vs Inference in Practice
Understanding the difference between training and inference is crucial for deploying LLMs in production financial systems. Let's see how this plays out in practice.
๐ Training: Fine-tuning on Financial Documents
Scenario: Fine-tune a large language model (70B+ params) on your bank's credit analysis reports
- โข Process thousands of examples in parallel (batch size 64-256)
- โข Each document teaches the model the pattern of credit analysis
- โข Model learns: high credit score + low DTI โ likely approval
- โข Training on 10,000 reports takes ~2-4 hours on A100 GPUs
๐ Inference: Real-time Credit Assessment
Scenario: Loan officer queries the fine-tuned model
Production Deployment Considerations for Financial Services
Key Takeaways: Training vs Inference
๐ Training
- โParallel processing of entire sequences
- โTeacher forcing with ground truth
- โFast and efficient (GPU saturated)
- โOne-time cost (expensive but amortized)
- โCausal mask prevents looking ahead
๐ Inference
- โSequential autoregressive generation
- โOne token at a time (cannot parallelize)
- โSlower (bottleneck for deployment)
- โPer-query cost (adds up at scale)
- โSampling strategy controls output quality
Why This Matters: Understanding training vs inference is essential for estimating costs, optimizing latency, and making architectural decisions when deploying LLMs in production financial systems. Training is a one-time investment; inference cost scales with usage.
Evolution of Foundation Models
From the first transformer in 2017 to today's multi-trillion parameter models, let's trace the remarkable evolution of foundation models and understand the key innovations at each stage.
GPT-1 (2018)
OpenAIKey Innovation: Generative Pre-training
- โPre-train on unlabeled text (BooksCorpus) using next-token prediction
- โFine-tune on specific tasks with small labeled datasets
- โShowed that unsupervised pre-training works better than training from scratch
- โEstablished the "pre-train then fine-tune" paradigm
BERT (2018)
GoogleKey Innovation: Bidirectional Training & Masked Language Modeling
- โMasked Language Modeling (MLM): Randomly mask 15% of tokens and predict them
- โBidirectional context: Can see both left and right context (unlike GPT)
- โNext Sentence Prediction (NSP): Learns relationships between sentences
- โDominated understanding tasks (classification, NER, Q&A) for years
The Debate: GPT (decoder-only, generative) vs BERT (encoder-only, understanding). Both architectures had their strengths - but the future belonged to decoder-only models...
GPT-2 (2019)
OpenAIKey Innovation: Scale + Zero-Shot Learning
- โ10ร larger than GPT-1, showing scaling laws in action
- โZero-shot capabilities: Could perform tasks without fine-tuning, just from prompts
- โCoherent text generation over longer contexts
- โSo good at generating realistic text that OpenAI initially delayed release due to safety concerns
GPT-3 (2020)
OpenAIKey Innovation: Emergent Abilities & Few-Shot Learning
- โ100ร larger than GPT-2 - the first truly "large" language model
- โIn-context learning: Could learn from examples in the prompt (few-shot)
- โEmergent abilities: Arithmetic, code generation, translation - not explicitly trained!
- โLaunched via API - democratized access to powerful AI
- โSparked the current Gen AI revolution and race
The Turning Point
GPT-3 proved that massive scale unlocks qualitatively new capabilities. The industry realized: bigger is genuinely better. The race to scale began.
ChatGPT / GPT-3.5 (2022)
OpenAIKey Innovation: RLHF (Reinforcement Learning from Human Feedback)
- โInstruction tuning: Trained to follow instructions and engage in dialogue
- โRLHF: Human feedback used to align model behavior with human preferences
- โConversational interface: Made AI accessible to everyone, not just developers
- โFastest-growing consumer application in history
- โBrought Gen AI into mainstream consciousness
The Inflection Point
ChatGPT made AI tangible for billions of people. Within weeks, every tech company pivoted to Gen AI. The era of ubiquitous AI assistants had begun.
LLaMA / LLaMA 2 / LLaMA 3 (2023-2024)
MetaKey Innovation: Open-Weight Efficiency
- โEfficiency focus: Competitive performance at smaller scales
- โOpen weights: Democratized access to powerful models
- โFine-tuning ecosystem: Spawned thousands of derivative models (Alpaca, Vicuna, etc.)
- โLLaMA 3.1 (405B) rivals GPT-4 on many benchmarks
- โEnabled on-premise deployment for regulated industries
Impact on Finance
Open-weight models like LLaMA are crucial for financial institutions that require data sovereignty, customization, and regulatory compliance. Banks can fine-tune on proprietary data without sending it to third parties.
GPT-4 (2023)
OpenAIKey Innovations: Multimodality + Reasoning + MoE
- โMultimodal: Processes images, text, and code seamlessly
- โAdvanced reasoning: Scores in 90th percentile on bar exam, excels at math
- โMoE architecture: Uses ~280B params per forward pass (efficient)
- โTool use: Can call functions, browse web, run code
- โMassive context window allows analyzing entire codebases or reports
State of the Art (2023-2024)
GPT-4 represented the frontier of LLM capabilities: reasoning, coding, multimodal understanding. It showed that scaling + architectural improvements continue to yield gains.
PaLM 2 & Gemini (2023-2024)
GoogleKey Innovations: Native Multimodality + Efficiency
- โPaLM 2: Excels at multilingual tasks, reasoning, and code
- โGemini Ultra: Competitive with GPT-4 on most benchmarks
- โNative multimodality: Trained on text, images, audio, video from the start
- โModel sizes: Nano, Pro, Ultra for different use cases
- โIntegrated deeply into Google ecosystem (Search, Workspace, Cloud)
Claude (2, 3, 3.5) (2023-2024)
AnthropicKey Innovations: Constitutional AI + Long Context + Reasoning
- โConstitutional AI: Trained with AI-generated feedback based on principles
- โExtended context: 200K tokens enables processing entire books
- โClaude 3.5 Sonnet: Best-in-class coding and reasoning
- โSafety focus: Lower refusal rates on harmless queries, higher on harmful ones
- โPreferred by many developers for nuanced, thoughtful responses
Why Claude Matters for Finance
Claude's extended context window and strong reasoning make it ideal for analyzing lengthy financial documents (10-Ks, contracts, regulations). Its safety-first approach aligns with compliance requirements.
The Current Landscape (2024-2025)
Key Trends
- โขMultimodality: Text + Vision + Audio + Video integration
- โขLong context: 100K-1M+ token windows
- โขEfficiency: Smaller models (7B-70B) approaching GPT-3.5 quality
- โขSpecialization: Domain-specific models for code, math, finance
- โขInference optimization: Quantization, distillation, speculative decoding
Open Questions
- โWill scaling laws continue indefinitely, or are we approaching limits?
- โHow do we verify outputs and prevent hallucinations at scale?
- โWhat's the right balance between open and closed models?
- โCan we achieve GPT-4 level performance at 10ร lower cost?
- โHow do we ensure safety, alignment, and regulatory compliance?
What's Next?
The next frontier: reasoning models (OpenAI o1, o3), agentic AI that can plan and execute complex tasks, personalized models that adapt to individual users, and multimodal agents that seamlessly interact with the physical and digital world. We're moving from passive question-answering to active problem-solving partners.
Fine-Tuning Large Language Models
Pre-trained foundation models are powerful, but they're generalists. To excel at specific tasks or domains (like financial analysis), we need to adapt them. Fine-tuning is the process of taking a pre-trained model and specializing it for your use case.
Why Fine-Tune?
Domain Expertise
Adapt to specialized domains (finance, medicine, law) with domain-specific knowledge and terminology
Better Performance
Achieve superior results on specific tasks compared to zero-shot or few-shot prompting
Data Privacy
Use proprietary data to customize models without exposing it via API prompts
Cost Efficiency
Smaller fine-tuned models can match larger base models, reducing inference costs
Fine-Tuning Approaches
Full Fine-Tuning
Update all parameters of the model during training on your custom dataset.
Advantages
- โMaximum performance gains
- โModel fully adapts to new domain
Disadvantages
- โExtremely expensive (compute & memory)
- โRisk of catastrophic forgetting
- โNeed multiple GPU clusters
PEFT (Parameter-Efficient Fine-Tuning)
Update only a small subset of parameters, keeping the base model frozen.
Advantages
- โTrain on consumer GPUs
- โ100-1000ร fewer params
- โNo catastrophic forgetting
- โSwap adapters for multi-task
Trade-offs
- โ Slightly lower performance ceiling
- โ Requires careful hyperparameter tuning
Supervised Fine-Tuning (SFT)
The most common form of fine-tuning: train the model on input-output pairs to teach it specific behaviors or knowledge.
SFT Process
Prepare Training Data
Create dataset of (instruction, response) pairs
Fine-Tune Model
Train using supervised learning (next-token prediction)
Model learns to generate outputs that match your training examples
Evaluate & Deploy
Test on held-out data, measure performance, deploy for inference
SFT for Financial Services
- โFine-tune on financial reports to improve 10-K/10-Q analysis
- โTrain on compliance documents to answer regulatory questions
- โUse customer service transcripts to build banking chatbots
- โAdapt to risk assessment tasks with historical loan data
RLHF: Reinforcement Learning from Human Feedback
RLHF is how models like ChatGPT learned to be helpful, harmless, and honest. Instead of just predicting the next token, models learn to maximize human preferences.
RLHF Pipeline
Phase 1: Supervised Fine-Tuning
Base โ SFT ModelStart with high-quality human demonstrations to teach basic instruction-following behavior.
Phase 2: Train Reward Model
Human RankingsTrain a reward model to predict which responses humans prefer. This model outputs a scalar score for any (prompt, response) pair.
Phase 3: RL Fine-Tuning (PPO)
Optimize for RewardUse PPO (Proximal Policy Optimization) to fine-tune the model to maximize the reward model's scores, while staying close to the original model.
The KL constraint prevents the model from drifting too far from its original behavior
RLHF Model
A model aligned with human preferences: helpful, honest, and harmless
Why RLHF Matters
- โMakes models conversational and instruction-following
- โReduces harmful outputs and improves safety
- โCan encode complex human values that are hard to specify with rules
- โUsed by ChatGPT, Claude, Gemini, and most production models
PEFT Techniques: Fine-Tuning on a Budget
Parameter-Efficient Fine-Tuning methods allow you to adapt large models with minimal compute. Instead of updating billions of parameters, update only millions.
LoRA (Low-Rank Adaptation)
Most PopularCore Idea
Instead of updating the full weight matrix W, add a low-rank decomposition: W' = W + ฮW = W + BA, where B and A are much smaller matrices.
Advantages
- โTrain 7B models on single consumer GPU
- โAdapter files are tiny (few MB vs many GB)
- โCan merge adapters or swap them on the fly
- โNo inference latency penalty (merge at load)
Common Use Cases
- โDomain adaptation (finance, medicine, legal)
- โTask-specific fine-tuning (summarization, QA)
- โMulti-tenant serving (one base model, many adapters)
QLoRA (Quantized LoRA)
Combines quantization (4-bit) with LoRA to enable fine-tuning even larger models on limited hardware.
Result: Fine-tune 65B models on a single 48GB GPU! Democratizes LLM customization.
Other PEFT Methods
Adapter Layers
Insert small trainable modules between frozen transformer layers. Simple but adds inference latency.
Prefix Tuning
Prepend trainable "virtual tokens" to the input. Model learns task-specific context without changing weights.
Prompt Tuning
Learn soft prompts (continuous embeddings) instead of discrete text. Extremely parameter-efficient.
Practical Considerations for Financial Institutions
When to Use Each Approach
You have labeled examples and want task-specific behavior
You need human-like judgment and alignment with preferences
Limited compute budget or need multiple task-specific models
Key Challenges
- โ Data quality: Garbage in, garbage out. Need high-quality training data.
- โ Overfitting: Small datasets can cause models to memorize instead of generalize
- โ Catastrophic forgetting: Fine-tuning can degrade general capabilities
- โ Evaluation: Hard to measure if model truly improved on your specific task
- โ Compliance: Need audit trails, version control, and model governance
Recommendation for Financial Institutions
Start with LoRA fine-tuning on open-weight models (LLaMA 3, Mistral). This gives you control, data privacy, and cost efficiency. Use SFT for task-specific adaptation, then consider RLHF only if you need sophisticated alignment with complex human preferences. Always maintain rigorous evaluation pipelines and model versioning for regulatory compliance.
Latest Advancements in Foundation Models
The foundation model landscape has shifted dramatically. Here is what every practitioner needs to know today.
The Reasoning Model Revolution
A new class of models โ called reasoning models โ emerged in late 2024 and changed how we think about model capabilities. Instead of answering immediately, they spend extra compute thinking before responding (test-time compute).
OpenAI's reasoning series. o3 achieves near-human performance on graduate-level science and advanced math. o4-mini is fast and cheap.
Anthropic's Claude 3.5/4.x with thinking mode: the model shows its reasoning chain. Excellent for multi-step compliance and legal analysis.
Open-weights Chinese reasoning model matching o1-level performance at a fraction of the cost. Shook the market in early 2025.
Google's reasoning model with 1M-token context. Strong on code, science, and multimodal reasoning.
Finance Impact
Reasoning models excel at multi-step financial analysis: "Given this 10-K, calculate the debt coverage ratio, compare it to the industry average, and flag any covenant risks." Tasks that used to require Chain-of-Thought prompting now just need a clear goal statement.
Context Window Explosion
Context windows have grown 100ร in two years, fundamentally changing what's possible without RAG.
| Model | Context Window | What Fits |
|---|---|---|
| GPT-3.5 (2022) | 4K tokens | ~3 pages |
| GPT-4 Turbo | 128K tokens | ~100 pages / a short book |
| Claude 3.5/4.x | 200K tokens | ~150 pages / a full annual report |
| Gemini 2.0/2.5 Flash | 1M tokens | ~750 pages / an entire regulatory handbook |
Implication: for many financial document tasks (single annual report Q&A, one regulatory circular), you can now load the entire document into context instead of building a RAG pipeline. RAG still wins for large corpora, freshness, access control, and cost at scale.
Model Landscape Update (2025โ2026)
Best for nuanced reasoning, safety-critical finance apps.
o3 for hard problems; o4-mini for cost-efficient reasoning at scale.
Best multimodal capabilities; Flash is very cost-efficient.
Open weights; deploy in your own infrastructure for data control.
Competitive with frontier models; open weights; cost disruption.
Small but capable; runs on-device; no data leaves the endpoint.
Small Language Models (SLMs)
Phi-4 (14B), Gemma 3 (27B), Llama 3.2 (3B/11B) prove you don't need a massive model for every task.
- โขRun entirely on-device (laptop, mobile, edge server)
- โขNo data leaves your environment โ critical for MNPI handling
- โขFaster and cheaper for high-volume, simple tasks
- โขFinance use: document classification, PII detection, form parsing
Multimodal Becomes Standard
Every frontier model now handles text + images natively. Some handle audio and video too.
- โขGPT-4o: real-time text + vision + audio in one model
- โขGemini 2.0: text + vision + audio + video + code execution
- โขFinance use: read earnings slides directly, analyze portfolio charts, process scanned regulatory forms
- โขNo more OCR pipelines for standard document types