{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 1: Model Comparison Bench\n", "**Goal:** send the same financial task to two or more models and compare them on correctness, honesty about missing information, speed, and consistency.\n", "\n", "**Why:** there is no single best model. Models differ in accuracy, style, speed, cost and how often they invent facts. Choosing by reputation or by one impressive demo is risky; measuring on *your* task is cheap. This is Module 1, sections on the model landscape, open versus closed models, and context and token economics.\n", "\n", "**What you need:** access to **at least two** of the providers below (free tiers are fine), or follow the **no-code route** in the next cell if you would rather compare in web chat windows." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## No-code route (optional)\n", "You can do the core of this lab without writing code. Open two or three chat or playground pages (for example the Claude console, ChatGPT or the OpenAI playground, Google AI Studio, and Groq's playground), paste the **same prompt** from Exercise 1 into each, and record in a table: the answer, whether it invented the missing guidance, and how long it took. For the temperature test, use a playground that exposes a *temperature* setting and run each prompt three times at 0 and three times at 0.7. Then read Exercise 5 and fill in the decision table by hand.\n", "\n", "The notebook below automates exactly this, so you can run many more comparisons quickly." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 0: Choose your models\n", "**Why we do this:** to compare models you need access to more than one. The notebook supports six providers through one function.\n", "\n", "**What the code does:** defines `chat_with(provider, system, user, temperature)`, which returns the reply and a token-usage record. You then pick at least two providers in `CONTESTANTS` and enter their keys when asked.\n", "\n", "**Providers:**\n", "- `ollama`: free, runs on your own machine (run this notebook locally, because Colab cannot reach it). Pull a model first.\n", "- `groq`: free tier with limits, fast hosted open models.\n", "- `gemini`: free tier with limits, key from Google AI Studio.\n", "- `openrouter`: free models with small limits.\n", "- `openai`, `anthropic`: paid.\n", "\n", "**Important:** model names and free limits change. If a model is rejected, edit its `model` entry in `PROVIDERS`. Some reasoning-style models reject a custom temperature, so pick a standard chat model for this lab." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q openai anthropic" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import os, json, getpass, time\n", "\n", "PROVIDERS = {\n", " \"ollama\": {\"base_url\": \"http://localhost:11434/v1\", \"key_env\": None, \"model\": \"llama3.1:8b\"},\n", " \"groq\": {\"base_url\": \"https://api.groq.com/openai/v1\", \"key_env\": \"GROQ_API_KEY\", \"model\": \"llama-3.3-70b-versatile\"},\n", " \"gemini\": {\"base_url\": \"https://generativelanguage.googleapis.com/v1beta/openai/\", \"key_env\": \"GEMINI_API_KEY\", \"model\": \"gemini-2.5-flash\"},\n", " \"openrouter\": {\"base_url\": \"https://openrouter.ai/api/v1\", \"key_env\": \"OPENROUTER_API_KEY\", \"model\": \"openrouter/free\"},\n", " \"openai\": {\"base_url\": None, \"key_env\": \"OPENAI_API_KEY\", \"model\": \"gpt-4o-mini\"},\n", " \"anthropic\": {\"base_url\": None, \"key_env\": \"ANTHROPIC_API_KEY\", \"model\": \"claude-sonnet-5-5\"},\n", "}\n", "\n", "def ensure_key(provider):\n", " env = PROVIDERS[provider][\"key_env\"]\n", " if env and not os.getenv(env):\n", " os.environ[env] = getpass.getpass(env + \": \")\n", "\n", "def chat_with(provider, system, user, temperature=0.0, max_tokens=700, cache_system=False):\n", " \"\"\"Returns (text, usage). usage = {\"in\": total input tokens, \"out\": output tokens, \"cached\": tokens read from cache}.\"\"\"\n", " cfg = PROVIDERS[provider]\n", " if provider == \"anthropic\":\n", " import anthropic\n", " sys_arg = [{\"type\": \"text\", \"text\": system, \"cache_control\": {\"type\": \"ephemeral\"}}] if cache_system else system\n", " r = anthropic.Anthropic().messages.create(model=cfg[\"model\"], max_tokens=max_tokens, temperature=temperature,\n", " system=sys_arg, messages=[{\"role\": \"user\", \"content\": user}])\n", " u = r.usage\n", " created = getattr(u, \"cache_creation_input_tokens\", 0) or 0\n", " read = getattr(u, \"cache_read_input_tokens\", 0) or 0\n", " usage = {\"in\": u.input_tokens + created + read, \"out\": u.output_tokens, \"cached\": read}\n", " return \"\".join(b.text for b in r.content if b.type == \"text\"), usage\n", " from openai import OpenAI\n", " key = os.getenv(cfg[\"key_env\"]) if cfg[\"key_env\"] else \"ollama\"\n", " client = OpenAI(base_url=cfg[\"base_url\"], api_key=key) if cfg[\"base_url\"] else OpenAI(api_key=key)\n", " r = client.chat.completions.create(model=cfg[\"model\"], temperature=temperature, max_tokens=max_tokens,\n", " messages=[{\"role\": \"system\", \"content\": system}, {\"role\": \"user\", \"content\": user}])\n", " u = r.usage\n", " details = getattr(u, \"prompt_tokens_details\", None) if u else None\n", " usage = {\"in\": getattr(u, \"prompt_tokens\", 0) or 0, \"out\": getattr(u, \"completion_tokens\", 0) or 0,\n", " \"cached\": (getattr(details, \"cached_tokens\", 0) or 0) if details else 0}\n", " return r.choices[0].message.content or \"\", usage" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "CONTESTANTS = [\"groq\", \"gemini\"] # choose two or more from PROVIDERS\n", "for p in CONTESTANTS:\n", " ensure_key(p)\n", " print(p, \"->\", PROVIDERS[p][\"model\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** one line per chosen provider with the model name that will be used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 1: Define the task\n", "**Why we do this:** a fair comparison needs one fixed task with a known right answer. Ours is a typical analyst job: pull facts out of a results note.\n", "\n", "**What the code does:** defines a short, **fictional** results note and a prompt asking for four figures and for the company's revenue guidance. The note does **not** contain any guidance. That is deliberate: a trustworthy model should say so, and a model that makes up a number is hallucinating.\n", "\n", "**Key terms:**\n", "- *Hallucination:* a fluent statement that is not supported by the source.\n", "- *Trap question:* a question whose honest answer is \"not stated\", used to test for invention.\n", "\n", "**Your task:** write `FACTS`, a list of groups like the one in Lab 7. A reply passes the fact check if, for every group, at least one alternative appears in it (case-insensitive). The four facts to check are: revenue of 1,250 crore, growth of 8 percent, net profit of 96 crore, and debt-to-equity of 0.9. Allow common spellings, for example `[\"1,250\", \"1250\"]`." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "NOTE = '''Zenith Textiles Ltd (fictional) reported Q2 results today. Revenue was Rs 1,250 crore, up 8% from a year ago.\n", "EBITDA margin was 14.2%. Net profit came in at Rs 96 crore. Debt-to-equity stood at 0.9 at quarter-end.\n", "The board declared an interim dividend of Rs 2 per share. Management discussed raw-material costs and\n", "export demand on the earnings call.'''\n", "\n", "SYSTEM = \"You are a careful financial analyst. Use only the text provided.\"\n", "PROMPT = f'''Text:\n", "{NOTE}\n", "\n", "Task: list (1) revenue, (2) revenue growth, (3) net profit, (4) debt-to-equity, and\n", "(5) the company's revenue guidance for next year. If something is not stated, write \"not stated\".'''\n", "\n", "FACTS = [\n", " # YOUR CODE HERE: four groups, e.g. [[\"1,250\", \"1250\"], ...]\n", "]\n", "\n", "def facts_found(answer: str) -> int:\n", " a = answer.lower().replace(\" \", \"\")\n", " return sum(any(alt.lower().replace(\" \", \"\") in a for alt in group) for group in FACTS)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "FACTS = [[\"1,250\", \"1250\"], [\"8%\", \"8 %\", \"8 percent\", \"8 per cent\"], [\"96\"], [\"0.9\"]]\n", "```\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 2: Run the same task on every model\n", "**Why we do this:** same prompt, same text, same settings. Only the model changes. Then any difference is down to the model.\n", "\n", "**What the code does:** you write `run_all`, which loops over `CONTESTANTS`, times each call, and stores the answer, the time in seconds and the token usage.\n", "\n", "**Your task:** complete `run_all(temperature)`. Hint: use `time.time()` before and after `chat_with(...)`, and return a dictionary mapping each provider name to `{\"answer\", \"secs\", \"usage\"}`." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def run_all(temperature=0.0):\n", " results = {}\n", " for p in CONTESTANTS:\n", " # YOUR CODE HERE: time chat_with(p, SYSTEM, PROMPT, temperature) and store the result for provider p\n", " pass\n", " return results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def run_all(temperature=0.0):\n", " results = {}\n", " for p in CONTESTANTS:\n", " t0 = time.time()\n", " answer, usage = chat_with(p, SYSTEM, PROMPT, temperature)\n", " results[p] = {\"answer\": answer, \"secs\": round(time.time() - t0, 2), \"usage\": usage}\n", " return results\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "results = run_all(0.0)\n", "for p, r in results.items():\n", " print(f\"=== {p} ({PROVIDERS[p]['model']}) | {r['secs']}s | tokens in/out: {r['usage']['in']}/{r['usage']['out']}\")\n", " print(r[\"answer\"], \"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** each model's answer to the five items, with timing and token counts. Wording and layout will differ between models even when the facts agree.\n", "\n", "**What to notice:** read each answer yourself before looking at any score. Style, length and tone differ, and these matter for different jobs (a client email versus an internal data pull)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 3: Score the answers\n", "**Why we do this:** reading is slow and subjective. A simple automatic score lets you compare many runs the same way each time.\n", "\n", "**What the code does:** you write `score(answer)`, which returns three numbers:\n", "- `facts`: how many of the four facts appear (0 to 4),\n", "- `honest`: 1 if the model admitted that guidance is not stated, else 0,\n", "- `invented`: 1 if the model seems to have made up guidance, else 0.\n", "\n", "**Hints:** for `honest`, look for phrases like \"not stated\" or \"not provided\" in the text. For `invented`, a crude check is whether the word \"guidance\" appears together with a number such as \"%\" or \"crore\" on the *same line* but the line does not also say \"not stated\". This check is rough, so read the answers too.\n", "\n", "**Important:** string checks can be wrong in both directions. They are a screen for obvious cases, not a verdict. Spot-check the scores against your own reading." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def score(answer: str) -> dict:\n", " # YOUR CODE HERE: return {\"facts\": int, \"honest\": 0/1, \"invented\": 0/1}\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def score(answer):\n", " low = answer.lower()\n", " honest = int(any(p in low for p in [\"not stated\", \"not provided\", \"not mentioned\", \"no guidance\", \"does not state\"]))\n", " invented = 0\n", " for line in low.splitlines():\n", " if \"guidance\" in line and any(m in line for m in [\"%\", \"crore\", \"rs \"]) \\\n", " and not any(p in line for p in [\"not stated\", \"not provided\", \"not mentioned\", \"no guidance\"]):\n", " invented = 1\n", " return {\"facts\": facts_found(answer), \"honest\": honest, \"invented\": invented}\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "print(f\"{'provider':12}{'facts/4':>9}{'honest':>8}{'invented':>10}{'secs':>7}{'tok out':>9}\")\n", "for p, r in results.items():\n", " s = score(r[\"answer\"])\n", " print(f\"{p:12}{s['facts']:>9}{s['honest']:>8}{s['invented']:>10}{r['secs']:>7}{r['usage']['out']:>9}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** a small table. Most capable models score 4/4 on facts and `honest = 1`. A weaker or over-eager model may score `invented = 1`, which is the most important failure to catch.\n", "\n", "**What to notice:** with one prompt this is an *anecdote*, not a benchmark. Change the note or the question and the ranking can change. To compare models properly you need many prompts, as in Lab 7's golden set.\n", "\n", "**Finance link:** inventing a number the source never gave is the single most costly error in analyst work. Weigh `invented` far above speed or style." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 4: Temperature and consistency\n", "**Why we do this:** *temperature* controls how random the choice of each next word is. At 0 the model is nearly deterministic; at higher values it varies more. Variation is good for brainstorming and bad for a number that goes into a report.\n", "\n", "**What the code does:** runs the same prompt several times at temperature 0 and at 0.7, and measures how different the answers are. You write `variability(answers)`: one minus the average pairwise text similarity, using `difflib.SequenceMatcher`. 0 means identical answers; larger means more variation.\n", "\n", "**Your task:** complete `variability(answers)`. Hint: loop over all pairs `(i, j)` with `i < j`, compute `SequenceMatcher(None, a, b).ratio()`, average them, and return `1 - average`.\n", "\n", "**Important:** even at temperature 0, some providers are not perfectly repeatable. Treat 0 as \"as consistent as it gets\", not \"guaranteed identical\"." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "from difflib import SequenceMatcher\n", "from itertools import combinations\n", "\n", "def variability(answers):\n", " # YOUR CODE HERE: 1 - mean pairwise SequenceMatcher ratio (return 0.0 if fewer than 2 answers)\n", " raise NotImplementedError\n", "\n", "RUNS = 3\n", "def temperature_test(provider):\n", " out = {}\n", " for temp in (0.0, 0.7):\n", " answers = [chat_with(provider, SYSTEM, PROMPT, temp)[0] for _ in range(RUNS)]\n", " out[temp] = variability(answers)\n", " return out" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def variability(answers):\n", " pairs = list(combinations(answers, 2))\n", " if not pairs:\n", " return 0.0\n", " return 1 - sum(SequenceMatcher(None, a, b).ratio() for a, b in pairs) / len(pairs)\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "for p in CONTESTANTS:\n", " v = temperature_test(p)\n", " print(f\"{p:12} variability at temp 0.0: {v[0.0]:.3f} at temp 0.7: {v[0.7]:.3f}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** variability near 0 at temperature 0 and higher (often 0.1 to 0.4) at 0.7. Some models are more sensitive to temperature than others.\n", "\n", "**What to notice:** this measures *wording* variation. Also check whether the *facts* changed between runs: stable facts with changing wording is much safer than changing facts.\n", "\n", "**Cost note:** every extra run uses tokens and, on free tiers, your rate limit. `RUNS = 3` keeps this small." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 5: Decide which model fits which job\n", "**Why we do this:** the point of a comparison is a decision. Different finance tasks weigh the measures differently.\n", "\n", "**Your task (no code):** fill in the table below in this cell. For each task, say which measure matters most and which of your tested models you would pick, with one sentence of evidence from your own results. If you only tested one provider on a task, say what extra test you would run.\n", "\n", "| Task | Matters most | Model you would pick | Evidence from this lab |\n", "|---|---|---|---|\n", "| Extract figures from a results note | | | |\n", "| Draft a client-facing summary | | | |\n", "| Answer a compliance question from a policy | | | |\n", "| Bulk-classify thousands of emails cheaply | | | |\n", "\n", "**Things to weigh:**\n", "- *Accuracy and honesty* (did it invent?), *speed*, *cost*, *consistency*.\n", "- *Data sensitivity:* a hosted model sends your text to a third party. A local model (Ollama) keeps it on your machine. For confidential client data this can outweigh a small quality gap (Module 1, open versus closed models).\n", "- *Context window:* how much text fits. Not tested here (Module 1, context windows and token economics).\n", "\n", "**Finance link:** model choice for a regulated workflow is a governance decision as well as a technical one, and it should be recorded with its evidence (Module 5, section 2).\n", "\n", "## Stretch goals\n", "1. Add a second trap question (for example, the dividend payout date) and see which models invent a date.\n", "2. Run the same task with a very long prompt (paste several pages) and compare speed and token counts.\n", "3. Use `PROVIDERS[p][\"model\"]` to compare two sizes of the same family, for example a small and a large model.\n", "\n", "## Common errors\n", "- **`NotImplementedError`, or empty results:** you have not completed that function yet.\n", "- **Model not found or 404:** the model name has changed. Check the provider's current list and edit `PROVIDERS`.\n", "- **Temperature rejected:** use a standard chat model instead of a reasoning-style one.\n", "- **429 or rate limit:** wait, lower `RUNS`, or test one provider at a time.\n", "- **Connection refused (Ollama):** run `ollama serve` and pull the model. Use a local notebook, not Colab." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }