{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 0: Document Preprocessor\n", "**Goal:** turn a real regulatory PDF into clean, structured text with entities and keywords, ready to feed into Lab 3.\n", "\n", "**Why:** every language-model system starts with text, and real documents are messy. If the text going in is broken, everything built on it (search, RAG, summaries) is worse. This lab uses the classic NLP toolkit from Module 0: cleaning, tokenisation, named-entity recognition and TF-IDF. **No API key and no paid service is needed.**\n", "\n", "**Documents:** two real, public SEBI documents: a circular on retail algorithmic trading (February 2025) and a board memo on AI tools (December 2024). The notebook downloads them for you." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 0: Setup\n", "**Why we do this:** we need a PDF reader, a language-processing library (spaCy), a machine-learning library for TF-IDF (scikit-learn) and a plotting library.\n", "\n", "**What the code does:** installs the packages, downloads spaCy's small English model (about 12 MB), and downloads the two PDFs into a `data/` folder.\n", "\n", "**Key terms:** *spaCy* = an open-source library that splits text into tokens and tags words with grammar and entity information." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q pypdf spacy scikit-learn matplotlib nltk" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import os, re, json, urllib.request\n", "import spacy.cli\n", "try:\n", " import en_core_web_sm # noqa: F401\n", "except ImportError:\n", " spacy.cli.download(\"en_core_web_sm\")\n", "import spacy\n", "nlp = spacy.load(\"en_core_web_sm\")\n", "\n", "SOURCES = {\n", " \"sebi_retail_algo_circular_feb2025.pdf\": \"https://www.sebi.gov.in/sebi_data/attachdocs/feb-2025/1738665456458.pdf\",\n", " \"sebi_ai_responsibility_memo_dec2024.pdf\": \"https://www.sebi.gov.in/sebi_data/meetingfiles/dec-2024/1735042007618_1.pdf\",\n", "}\n", "os.makedirs(\"data\", exist_ok=True)\n", "for name, url in SOURCES.items():\n", " if not os.path.exists(f\"data/{name}\"):\n", " req = urllib.request.Request(url, headers={\"User-Agent\": \"Mozilla/5.0\"})\n", " open(f\"data/{name}\", \"wb\").write(urllib.request.urlopen(req).read())\n", "print(os.listdir(\"data\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** a list with the two PDF file names." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 1: Load and clean the text\n", "**Why we do this:** text pulled from a PDF is not the text you see on screen. It contains page headers, bullet symbols that come out as strange characters, curly quotes, and broken line breaks. Cleaning makes later steps reliable (Module 0, section 2).\n", "\n", "**What the code does:** `load_pages` reads each page into a list of `(page_number, raw_text)`. First look at the raw text and count the odd characters. Then you write `clean_text`.\n", "\n", "**Your task:** complete `clean_text(text)` so it:\n", "1. removes page markers such as `Page 3 of 6`,\n", "2. replaces the bullet character `\\uf0b7` with `- `,\n", "3. replaces curly quotes with straight quotes (`\\u201c \\u201d` become `\"` and `\\u2018 \\u2019` become `'`),\n", "4. collapses all runs of whitespace (including line breaks) into a single space.\n", "\n", "**Important:** some words in these PDFs are split by the extraction itself (for example \"mech anisms\"). Simple cleaning cannot fix that reliably. Note it and move on; it is a known limit of PDF text extraction." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from pypdf import PdfReader\n", "from collections import Counter\n", "\n", "def load_pages(path):\n", " return [(i + 1, p.extract_text() or \"\") for i, p in enumerate(PdfReader(path).pages)]\n", "\n", "RAW = {name: load_pages(f\"data/{name}\") for name in SOURCES}\n", "sample = RAW[\"sebi_retail_algo_circular_feb2025.pdf\"][1][1]\n", "print(repr(sample[:300]))\n", "\n", "all_raw = \" \".join(t for pages in RAW.values() for _, t in pages)\n", "odd = Counter(ch for ch in all_raw if ord(ch) > 127)\n", "print({hex(ord(ch)): n for ch, n in odd.most_common(6)})\n", "\n", "def clean_text(text: str) -> str:\n", " # YOUR CODE HERE: apply the four cleaning steps above and return the cleaned string.\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def clean_text(text):\n", " text = re.sub(r\"Page \\d+ of \\d+\", \" \", text)\n", " text = text.replace(\"\\uf0b7\", \"- \")\n", " text = text.replace(\"\\u201c\", '\"').replace(\"\\u201d\", '\"').replace(\"\\u2018\", \"'\").replace(\"\\u2019\", \"'\")\n", " return re.sub(r\"\\s+\", \" \", text).strip()\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "CLEAN = {name: [(pg, clean_text(t)) for pg, t in pages] for name, pages in RAW.items()}\n", "cleaned = \" \".join(t for pages in CLEAN.values() for _, t in pages)\n", "print(cleaned[:400])\n", "print(\"\\n'Page x of y' left:\", len(re.findall(r\"Page \\d+ of \\d+\", cleaned)), \"| bullet chars left:\", cleaned.count(\"\\uf0b7\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** a readable opening paragraph with no line breaks, and `0` for both leftovers. The raw sample shows `\\n` characters and page markers; the cleaned text should not.\n", "\n", "**What to notice:** the odd-character counter shows exactly what needed fixing. Always inspect your data before writing cleaning rules, instead of guessing.\n", "\n", "**Finance link:** a stray bullet or quote character can break a table lookup or a search for a clause. Clean input is the cheapest way to reduce downstream errors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 2: Tokenise and normalise\n", "**Why we do this:** computers need text split into units and reduced to a standard form so that \"regulation\", \"regulations\" and \"regulated\" can be treated as related (Module 0, section 2).\n", "\n", "**What the code does:** spaCy turns a sentence into *tokens*. Each token carries its *lemma* (dictionary form) and flags such as \"is a stop word\". We also compare it with *stemming*, a cruder method that chops word endings.\n", "\n", "**Key terms:**\n", "- *Token:* one unit of text, usually a word or punctuation mark.\n", "- *Stop word:* a very common word (\"the\", \"of\") that carries little meaning.\n", "- *Lemma:* the dictionary form of a word (\"brokers\" becomes \"broker\").\n", "- *Stem:* a chopped-off root that may not be a real word (\"regulation\" becomes \"regul\").\n", "\n", "**Your task:** complete `normalise(text)`. It should return a list of lowercase lemmas, keeping only alphabetic tokens that are not stop words. Hint: loop over `nlp(text)` and use `tok.lemma_`, `tok.is_alpha`, `tok.is_stop`." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from nltk.stem import PorterStemmer\n", "stemmer = PorterStemmer()\n", "\n", "doc = nlp(\"Brokers shall tag all algo orders approved by the Exchanges.\")\n", "print([t.text for t in doc])\n", "print(f\"{'word':12}{'lemma':12}{'stem':10}\")\n", "for w in [\"regulations\", \"brokers\", \"trading\", \"approved\", \"responsibility\"]:\n", " t = nlp(w)[0]\n", " print(f\"{w:12}{t.lemma_:12}{stemmer.stem(w):10}\")\n", "\n", "def normalise(text: str):\n", " # YOUR CODE HERE: return lowercase lemmas of alphabetic, non-stop-word tokens.\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def normalise(text):\n", " return [t.lemma_.lower() for t in nlp(text) if t.is_alpha and not t.is_stop]\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "page = CLEAN[\"sebi_retail_algo_circular_feb2025.pdf\"][0][1]\n", "terms = normalise(page)\n", "print(len(page.split()), \"words ->\", len(terms), \"normalised terms\")\n", "print(terms[:25])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** a word count dropping to roughly half as many terms, and a list of lowercase content words such as `circular`, `algorithmic`, `retail`, `investor`. The exact list depends on the spaCy model version.\n", "\n", "**What to notice:** in the lemma and stem table, lemmas are always real words and stems are not. Stemming is faster but rougher; lemmatisation is usually better for readable results." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 3: Find entities\n", "**Why we do this:** extracting *who, when, how much and which rule* turns paragraphs into data you can search and check (Module 0, section 4).\n", "\n", "**What the code does:** two approaches side by side.\n", "- *Statistical NER (spaCy):* a trained model labels organisations, dates, money and so on. It handles varied wording but can miss domain-specific items.\n", "- *Rules (regular expressions):* a pattern you write. It is exact for formats you know, such as circular numbers like `CIR/MRD/DP/09/2012`, but only finds what you describe.\n", "\n", "**Key terms:** *NER* = named-entity recognition. *Regular expression* = a compact pattern for matching text shapes.\n", "\n", "**Your task:** complete `regex_entities(text)` to return a dictionary with three lists: `circular_refs` (slash-separated reference numbers), `dates` (like `February 4, 2025`) and `amounts` (rupee amounts such as `Rs. 5 crore` or `\\u20b91,250 crore`). Patterns are in the hints below." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from collections import Counter\n", "\n", "text = \" \".join(t for _, t in CLEAN[\"sebi_retail_algo_circular_feb2025.pdf\"])\n", "ents = nlp(text[:6000]).ents\n", "print(Counter(e.label_ for e in ents).most_common())\n", "print([e.text for e in ents if e.label_ == \"ORG\"][:8])\n", "print([e.text for e in ents if e.label_ == \"DATE\"][:8])\n", "\n", "MONTHS = \"January|February|March|April|May|June|July|August|September|October|November|December\"\n", "\n", "def regex_entities(text: str) -> dict:\n", " # YOUR CODE HERE: return {\"circular_refs\": [...], \"dates\": [...], \"amounts\": [...]}\n", " # circular_refs hint: r\"\\b[A-Z]{2,}(?:/[A-Za-z0-9\\-]+){2,}\\b\"\n", " # dates hint: rf\"(?:{MONTHS})\\s+\\d{{1,2}},?\\s+\\d{{4}}\"\n", " # amounts hint: r\"(?:Rs\\.?|INR|\\u20b9)\\s?[\\d,]+(?:\\.\\d+)?(?:\\s?(?:crore|lakh|million|billion))?\"\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def regex_entities(text):\n", " return {\n", " \"circular_refs\": re.findall(r\"\\b[A-Z]{2,}(?:/[A-Za-z0-9\\-]+){2,}\\b\", text),\n", " \"dates\": re.findall(rf\"(?:{MONTHS})\\s+\\d{{1,2}},?\\s+\\d{{4}}\", text),\n", " \"amounts\": re.findall(r\"(?:Rs\\.?|INR|\\u20b9)\\s?[\\d,]+(?:\\.\\d+)?(?:\\s?(?:crore|lakh|million|billion))?\", text),\n", " }\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "found = regex_entities(text)\n", "print(found[\"circular_refs\"])\n", "print(found[\"dates\"])\n", "print(regex_entities(\"The fee is Rs. 5 crore and the limit is \\u20b91,250 crore.\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** circular references including `SEBI/HO/MIRSD/MIRSD-PoD/P/CIR/2025/0000013` and `CIR/MRD/DP/09/2012`, a list of dates such as `February 4, 2025` and `March 30, 2012`, and the two rupee amounts from the test sentence. (The circulars themselves contain few rupee amounts, which is why we also test on a sentence.)\n", "\n", "**What to notice:** compare the dates from spaCy with those from your pattern. Rules are precise for known formats; the model is more flexible. Real pipelines use both, and the circular reference numbers are something no general model will recognise.\n", "\n", "**Finance link:** pulling out amounts, dates and reference numbers is the first step of KYC checks, loan-document review and regulatory tracking (Module 0, section 7)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 4: TF-IDF, which terms are distinctive?\n", "**Why we do this:** a word that appears everywhere (\"SEBI\", \"shall\") tells you little about one page. A word that appears often on one page and rarely elsewhere is *distinctive*. TF-IDF scores exactly that (Module 0, section 3).\n", "\n", "**What the code does:** treats each page as a small document, then scores every term. *TF* (term frequency) rewards words repeated on a page. *IDF* (inverse document frequency) penalises words found on many pages. The product is the TF-IDF score.\n", "\n", "**Key terms:** *Vectoriser* = a tool that turns text into numeric scores. *Bigram* = a two-word phrase such as \"algo provider\".\n", "\n", "**Your task:** complete `top_terms(row, n)`. Hint: the vectoriser gives a matrix; take one row, sort its scores from high to low, and look up the term names with `vectorizer.get_feature_names_out()`." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "\n", "PAGES = [(name, pg, t) for name, pages in CLEAN.items() for pg, t in pages if len(t) > 200]\n", "vectorizer = TfidfVectorizer(stop_words=\"english\", ngram_range=(1, 2), lowercase=True)\n", "matrix = vectorizer.fit_transform([t for _, _, t in PAGES])\n", "terms_index = vectorizer.get_feature_names_out()\n", "print(len(PAGES), \"pages,\", len(terms_index), \"distinct terms\")\n", "\n", "def top_terms(row: int, n: int = 8):\n", " # YOUR CODE HERE: return a list of (term, score) for the n highest-scoring terms on page `row`.\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def top_terms(row, n=8):\n", " scores = matrix[row].toarray().ravel()\n", " best = np.argsort(scores)[::-1][:n]\n", " return [(terms_index[i], round(float(scores[i]), 3)) for i in best]\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "for row in (0, 3, len(PAGES) - 1):\n", " name, pg, _ = PAGES[row]\n", " print(f\"{name} p.{pg}:\", [t for t, _ in top_terms(row)])\n", "\n", "terms, scores = zip(*top_terms(0, 8))\n", "plt.barh(terms[::-1], scores[::-1])\n", "plt.title(\"Most distinctive terms: first page\"); plt.xlabel(\"TF-IDF score\"); plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** a short list of distinctive terms per page and a bar chart. Terms specific to a page's topic should rank high, while terms used on every page should not.\n", "\n", "**What to notice:** the same word can be common on one page and unremarkable overall. With only about a dozen small documents the scores are rough; TF-IDF gets more reliable with more text." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 5: Produce structured output for Lab 3\n", "**Why we do this:** the point of preprocessing is to hand later steps something clean and structured. Here you combine everything into one JSON file with a record per page.\n", "\n", "**What the code does:** you write `process_page`, which returns a record with the document name, page number, cleaned text, entities (from spaCy and from your rules) and keywords (from TF-IDF). The cell then saves all records to `processed_documents.json`.\n", "\n", "**Your task:** complete `process_page(row)`. Hint: take `name, pg, t = PAGES[row]`, run `nlp(t)` for entities, call `regex_entities(t)` and `top_terms(row)`.\n", "\n", "**How Lab 3 uses this:** Lab 3 reads PDFs directly. As a stretch, change its loader to read the `text` field from this file and compare retrieval quality on cleaned versus raw text." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def process_page(row: int) -> dict:\n", " # YOUR CODE HERE: return {\"doc\", \"page\", \"text\", \"entities\", \"keywords\"} for page `row`.\n", " # entities should be a dict: {\"spacy\": [{\"text\":..., \"label\":...}, ...], \"rules\": regex_entities(t)}\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def process_page(row):\n", " name, pg, t = PAGES[row]\n", " return {\n", " \"doc\": name, \"page\": pg, \"text\": t,\n", " \"entities\": {\n", " \"spacy\": [{\"text\": e.text, \"label\": e.label_} for e in nlp(t).ents],\n", " \"rules\": regex_entities(t),\n", " },\n", " \"keywords\": [term for term, _ in top_terms(row)],\n", " }\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "records = [process_page(i) for i in range(len(PAGES))]\n", "with open(\"processed_documents.json\", \"w\", encoding=\"utf-8\") as f:\n", " json.dump(records, f, ensure_ascii=False, indent=2)\n", "print(len(records), \"records saved to processed_documents.json\")\n", "r = records[0]\n", "print(r[\"doc\"], \"p.\", r[\"page\"], \"| keywords:\", r[\"keywords\"][:5], \"| circular refs:\", r[\"entities\"][\"rules\"][\"circular_refs\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** a record count equal to the number of pages kept, and a sample record with keywords and the circular reference found on page 1.\n", "\n", "**What to notice:** open `processed_documents.json`. Every field came from a method you used in this lab. This is the shape of data that search, RAG and review workflows prefer to work with.\n", "\n", "## Stretch goals\n", "1. Fix the broken words (\"mech anisms\") by checking joined fragments against spaCy's vocabulary.\n", "2. Add a rule that finds regulation names such as \"Intermediaries Regulations, 2008\".\n", "3. Compare TF-IDF keywords with the keywords an LLM suggests for the same page. Which are more useful for search?\n", "\n", "## Common errors\n", "- **`NotImplementedError`:** you have not completed that exercise's function yet.\n", "- **spaCy model not found:** re-run the setup cell, then restart the runtime.\n", "- **A download fails (SEBI site):** retry, or download the PDFs in your browser and place them in `data/` under the same names.\n", "- **Entity lists look different from the page:** spaCy models are statistical and change between versions. That is normal." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }