From text to understanding - the bridge to modern AI. Learn the fundamentals of Natural Language Processing that laid the groundwork for today's powerful language models.
Natural Language Processing (NLP) is the field of AI that enables computers to understand, interpret, and generate human language. Before we had powerful foundation models like GPT and Claude, NLP researchers spent decades developing techniques to make sense of text.
NLP sits at the intersection of computer science, linguistics, and artificial intelligence. Its goal: enable machines to process and understand human language in a way that is both meaningful and useful.
Extracting meaning from text. Examples: sentiment analysis, named entity recognition, question answering.
Producing human-like text. Examples: report generation, summarization, machine translation.
Conversational AI. Examples: chatbots, virtual assistants, dialogue systems.
Human language is incredibly complex. Unlike structured data or images, language has ambiguity, context-dependency, and cultural nuance that make it challenging for machines to understand.
The same word can have different meanings depending on context:
Meaning changes based on surrounding words:
"The company reported strong earnings, but investors remain cautious."
→ Positive fundamentals, negative sentiment. Understanding requires reading the full sentence.
Literal meaning ≠ intended meaning:
"Oh great, another market crash. Just what we needed!" 😒
→ Clearly negative, but uses the word "great" (positive). Machines struggle with this.
Financial jargon requires specialized understanding:
"The Fed's hawkish stance on inflation has led to a flight to quality, with investors rotating from growth to value stocks."
→ Requires knowledge of monetary policy, market dynamics, and investment strategies.
NLP has gone through several paradigm shifts over the past 60+ years. Understanding this evolution helps us appreciate why modern foundation models are so powerful.
Hand-crafted rules and expert systems. Linguists manually wrote grammar rules and patterns.
Example: "If sentence contains 'not' before a positive word, classify as negative."
⚠️ Problem: Doesn't scale. Too many exceptions and edge cases.
Machine learning on large text corpora. Models learned patterns from data using probabilistic methods (Hidden Markov Models, Naive Bayes, etc.).
Example: "In financial news, 'earnings beat' appears 87% of the time with positive sentiment."
✓ Better generalization, but still limited by feature engineering.
Deep learning with word embeddings (Word2Vec, GloVe) and neural networks (RNNs, LSTMs). Models learned to represent words as dense vectors.
Example: "Revenue" and "sales" have similar vector representations because they appear in similar contexts.
✓ Automatic feature learning, but still struggled with long-range dependencies.
Attention mechanisms and Transformers revolutionized NLP. Models can attend to any part of the input, handling long-range dependencies effortlessly.
Example: BERT, GPT, Claude - models that understand context across entire documents.
✓ State-of-the-art across all NLP tasks. This is where we are today.
Finance has unique NLP challenges that make it particularly interesting (and difficult):
"$1.2B revenue, up 15% YoY" - numbers carry critical meaning
Dense, formal, precise language with legal implications
P/E ratios, EBITDA, alpha, beta - specialized vocabulary
"Fed raised rates" - when? Pre-2022 or post-2022? Huge difference.
Before we can analyze text, we need to clean and transform it into a format machines can process. This step is critical - garbage in, garbage out!
Text preprocessing involves cleaning and normalizing raw text to make it suitable for analysis. The goal: reduce noise and standardize the input.
Lowercasing
"Apple" and "apple" should be treated the same (usually)
text = text.lower() # "Apple Inc." → "apple inc."Removing Punctuation
Strip special characters (but be careful with financial data!)
import re
text = re.sub(r'[^\w\s]', '', text)Removing Stopwords
Remove common words like "the", "is", "and" (low information)
from nltk.corpus import stopwords
stops = set(stopwords.words('english'))Normalization
Handle contractions, unicode, special characters
"won't" → "will not" | "café" → "cafe"Tokenization is the process of splitting text into smaller units called tokens (words, subwords, or characters). This is the first step in any NLP pipeline.
Input:
"Apple Inc.'s Q4 revenue reached $90.1B, up 8% YoY."Word Tokenization:
['Apple', 'Inc', '.', "'s", 'Q4', 'revenue', 'reached', '$', '90.1', 'B', ',', 'up', '8', '%', 'YoY', '.']Split by whitespace and punctuation
Example:
"Hello world"
→ ["Hello", "world"]✓ Simple, interpretable
✗ Large vocabulary, out-of-vocabulary words
Split into smaller meaningful units (BPE, WordPiece)
Example:
"unhappiness"
→ ["un", "happiness"]✓ Handles rare words, smaller vocab
~ Used by GPT, BERT
Split into individual characters
Example:
"Hi"
→ ["H", "i"]✓ No out-of-vocabulary issues
✗ Very long sequences
Both techniques reduce words to their base form, but they work differently:
Crude heuristic process that chops off word endings. Fast but imprecise.
Examples:
"running" → "run"
"runner" → "run"
"running" → "run"
"studies" → "studi" ⚠️
"studying" → "study"Popular Algorithm: Porter Stemmer
Uses vocabulary and morphological analysis. Returns actual dictionary words.
Examples:
"running" → "run"
"better" → "good"
"was" → "be"
"studies" → "study" ✓
"studying" → "study"Popular Tool: spaCy, WordNet Lemmatizer
Financial text has special considerations - you can't just apply standard preprocessing blindly!
"$1.2B" and "15%" carry critical meaning in finance. Instead, normalize them: "$1.2B" → "MONEY_1.2B" or keep as-is.
"not" is a stopword, but "not profitable" vs "profitable" have opposite meanings! Consider context before removing.
Company names, tickers, dates, amounts - these are gold. Consider using Named Entity Recognition (NER) to protect them during preprocessing.
Example: Financial Text Preprocessing Pipeline
import re
from spacy import load
nlp = load("en_core_web_sm")
def preprocess_financial_text(text):
# 1. Normalize currency
text = re.sub(r'\$(\d+\.?\d*)(B|M|K)?', r'MONEY_\1\2', text)
# 2. Normalize percentages
text = re.sub(r'(\d+\.?\d*)%', r'PERCENT_\1', text)
# 3. Tokenization with spaCy (preserves entities)
doc = nlp(text)
# 4. Lemmatize but keep financial terms
tokens = [token.lemma_ for token in doc if not token.is_stop]
return tokensMachines only understand numbers. To process text, we need to represent words as numerical vectors. The question is: how do we capture meaning in numbers?
Bag of Words represents text as a vector of word counts. Simple, but loses word order and context.
Sentence 1: "Company revenue increased significantly"
Sentence 2: "Revenue increased slightly"
Vocabulary: [company, revenue, increased, significantly, slightly]
Sentence 1: [1, 1, 1, 1, 0]
Sentence 2: [0, 1, 1, 0, 1]Each position represents a word from the vocabulary. The number is the count of that word in the sentence.
✓ Advantages
✗ Disadvantages
TF-IDF (Term Frequency-Inverse Document Frequency) improves on BoW by giving more weight to rare, informative words and less weight to common words.
TF-IDF(word, document) = TF(word, document) × IDF(word)
TF = (# times word appears in doc) / (# total words in doc)
IDF = log(# total documents / # documents containing word)Intuition: If a word appears often in one document but rarely across all documents, it's probably important to that specific document.
Analyzing 3 earnings call transcripts:
Document 1 (Tech Company):
"Our cloud revenue grew 40% while AI investments increased."
Document 2 (Retail Company):
"Revenue grew 5% driven by store expansion."
Document 3 (Bank):
"Revenue grew 3% as loan volumes increased."
TF-IDF Scores:
"revenue" (appears in all 3) → Low TF-IDF (common)
"cloud" (only in Doc 1) → High TF-IDF (distinctive)
"store" (only in Doc 2) → High TF-IDF (distinctive)
"loan" (only in Doc 3) → High TF-IDF (distinctive)→ TF-IDF helps identify what makes each company unique!
BoW and TF-IDF treat words as independent units. Word embeddings represent words as dense vectors where similar words have similar vectors.
"You shall know a word by the company it keeps"
Words that appear in similar contexts tend to have similar meanings. Embeddings learn these relationships from data.
Example 1:
"The company reported strong earnings."
"The firm reported strong earnings."
→ "company" and "firm" appear in similar contexts
Example 2:
"Revenue increased by 10%."
"Sales increased by 10%."
→ "revenue" and "sales" are interchangeable
Predicts context words from target word (Skip-gram) or target from context (CBOW). Learns 100-300 dimensional vectors.
Famous example:
king - man + woman ≈ queen
Paris - France + Germany ≈ Berlin→ Vector arithmetic captures relationships!
Global Vectors. Uses word co-occurrence statistics. Pre-trained on Wikipedia, Common Crawl.
Financial example:
revenue - sales ≈ 0
(very similar vectors)
revenue - expense ≠ 0
(different concepts)→ Captures semantic relationships
Word2Vec and GloVe give each word a single, fixed vector regardless of context. This is a problem!
Sentence 1: "The bank approved my loan." (financial institution)
Sentence 2: "I sat by the river bank." (riverbank)
→ Word2Vec/GloVe assign the SAME vector to both uses of "bank"! 😕
This is where contextual embeddings (BERT, ELMo) and eventually Transformers came in! We'll explore this in Module 1. 🚀
NLP encompasses many different tasks, each solving a specific problem. Understanding these tasks helps you recognize when and how to apply NLP in finance.
Named Entity Recognition identifies and classifies named entities (people, organizations, locations, dates, amounts) in text.
Input Text:
"JPMorgan Chase CEO Jamie Dimon announced a $2.5 billion investment in AI technology on January 15, 2024."
NER Output:
Sentiment Analysis determines the emotional tone of text: positive, negative, or neutral.
✓ Positive
"Strong earnings beat expectations, revenue up 25%"
Score: +0.85
○ Neutral
"The company maintained its dividend at $0.50 per share"
Score: 0.00
✗ Negative
"Profit warnings issued as costs spiral out of control"
Score: -0.72
📰 News Sentiment
Track sentiment of news articles about stocks to predict price movements
📊 Earnings Call Analysis
Analyze tone of management commentary to gauge confidence
💬 Social Media Monitoring
Track public sentiment on Twitter/Reddit for trading signals
📝 Analyst Reports
Quantify bullish/bearish tone in research reports
Text Classification assigns predefined categories to documents.
"This annual report provides information about our business, risk factors, and financial condition..."
"Current report: Material event disclosure regarding acquisition of XYZ Corp..."
"Registration statement for initial public offering. We intend to offer..."
→ Automatically route documents to appropriate teams based on type
Question Answering (QA) systems extract answers from text given a question.
Context (from 10-K):
"Our total revenue for fiscal year 2023 was $125.3 billion, representing a 12% increase from the prior year. Operating income reached $45.2 billion with an operating margin of 36%."
Q: What was the total revenue in 2023?
A: $125.3 billion
Q: What was the operating margin?
A: 36%
→ Modern QA systems power chatbots that can answer questions about financial documents!
Before deep learning dominated NLP, traditional ML algorithms (Naive Bayes, SVM, Logistic Regression) were the go-to methods. They still have their place today!
Traditional ML requires manual feature engineering - converting text into numerical features that capture important patterns.
Probabilistic classifier based on Bayes' theorem. Assumes features are independent (naive assumption).
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import TfidfVectorizer
# Vectorize text
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(documents)
# Train classifier
clf = MultinomialNB()
clf.fit(X, labels) # labels: [0=negative, 1=positive]✓ Fast, works well with small datasets, good for spam detection and sentiment analysis
Finds the optimal hyperplane that separates classes with maximum margin. Powerful for high-dimensional data (like text).
from sklearn.svm import LinearSVC
# SVM with linear kernel
clf = LinearSVC()
clf.fit(X_train, y_train)
predictions = clf.predict(X_test)✓ Effective for text classification, especially with TF-IDF features
Despite the name, it's a classification algorithm. Models probability of class membership.
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
# Get probability scores
probabilities = clf.predict_proba(X_test)✓ Interpretable, outputs probabilities, good baseline for text classification
Deep learning isn't always the answer! Traditional ML still shines in certain scenarios:
Less than 10,000 examples? Traditional ML often outperforms deep learning which needs more data.
Can inspect feature weights to understand why model made a decision. Critical for regulated finance.
Fast inference, no GPU required. Can run on edge devices or in real-time systems.
Spam detection, basic sentiment - don't need a sledgehammer for a nail.
The 2010s brought a revolution: deep learning. Neural networks, especially those designed for sequences (RNNs, LSTMs), transformed NLP. Then came attention mechanisms - the breakthrough that led to Transformers and modern LLMs.
Unlike traditional neural networks that process fixed inputs, RNNs can handle sequences of variable length by maintaining a "memory" of previous inputs.
At each time step, an RNN takes:
Input: "Revenue", "increased", "by", "25%"
Step 1: "Revenue" → h1 (hidden state 1)
Step 2: "increased" + h1 → h2
Step 3: "by" + h2 → h3
Step 4: "25%" + h3 → h4 → Final prediction→ The hidden state "remembers" earlier words in the sentence!
RNNs struggled with long-range dependencies. By the time the network processed word #50, it had "forgotten" word #1.
Example: "The company, which was founded in 1995 and operates in 47 countries, reported strong earnings."
→ RNN struggles to connect "company" with "reported" due to long distance.
LSTMs solved the vanishing gradient problem with a more sophisticated memory mechanism: gates that control what to remember and forget.
Decides what information to discard from cell state
Decides what new information to add to cell state
Decides what to output based on cell state
✓ LSTMs could handle longer sequences, powering machine translation, speech recognition, and more!
Attention (2014-2017) was the breakthrough that revolutionized NLP. Instead of compressing the entire input sequence into a fixed vector, attention lets the model focus on relevant parts of the input dynamically.
When translating "The company reported strong revenue growth":
Attention Weights Visualization:
When generating "reported":
"The" → 0.05
"company" → 0.40 ← high attention
"reported" → 0.45 ← high attention
"strong" → 0.05
"revenue" → 0.03
"growth" → 0.02→ Attention solved the long-range dependency problem and became the foundation for Transformers (which we'll cover in Module 1)!
Let's see how NLP techniques we've learned are applied in finance today. These are production systems running at major financial institutions!
Problem: Analyzing thousands of earnings call transcripts manually is impossible. Can we automatically detect management sentiment?
NLP Pipeline:
✓ Hedge funds use this to make trading decisions within minutes of earnings releases!
Problem: Banks receive thousands of documents daily (contracts, invoices, regulatory filings). How to route them to the right department?
NLP Solution:
✓ Reduces manual processing time by 80%!
Problem: Monitor news and communications for mentions of sanctioned entities or potential money laundering patterns.
NLP Techniques:
→ Critical for AML (Anti-Money Laundering) and KYC (Know Your Customer) compliance
Problem: Customers ask the same questions repeatedly. Can we automate responses while maintaining accuracy?
Modern Approach:
✓ 60-70% of customer queries can be automated with high accuracy!
We've covered the evolution of NLP from simple Bag of Words to sophisticated attention mechanisms. Now let's connect this to the Transformer revolution that birthed modern foundation models.
Every task (sentiment, NER, classification) required training a separate model from scratch. No knowledge transfer.
Even LSTMs struggled with paragraphs or documents. Context window was limited.
Required thousands of labeled examples for each new task. Expensive and time-consuming.
RNNs/LSTMs processed words one at a time. Couldn't be parallelized. Slow training on long sequences.
The Transformer architecture, introduced by Google researchers in 2017, changed everything. It threw away recurrence entirely and relied purely on attention mechanisms.
Each word attends to every other word in the sentence, learning which words are most relevant to each other.
"The bank approved my loan."
→ "bank" attends strongly to "approved" and "loan" to understand it's a financial institution, not a riverbank.
Transformer (Google)
Original architecture for machine translation. "Attention Is All You Need" paper.
BERT (Google) & GPT-1 (OpenAI)
First pre-trained Transformer models. BERT: bidirectional, GPT: autoregressive (predict next word).
GPT-2, GPT-3
Scaling up: 1.5B → 175B parameters. Few-shot learning emerged. "Foundation model" era begins.
ChatGPT, GPT-4, Claude, Gemini
Instruction-tuned, RLHF-aligned models. Multimodal capabilities. The era we're in now!
You've now learned the foundations of NLP: from basic text preprocessing to sophisticated neural architectures. You understand what problems Transformers solved and why modern foundation models are so powerful.
In Module 1, we'll dive deep into Foundation Models themselves - GPT, Claude, BERT - and understand how they're revolutionizing finance. Let's go! 🚀