Observability & Evaluation Stack

Why we do this: The same provider setup as Labs 4 and 6. This lab makes about 40 model calls per run (20 answers and 20 judgements), so a free tier with tight limits may slow you down.

What the code does: Defines chat(messages), then an ask(system, user) helper with a few retries and a short wait, because rate-limit errors are common when many calls run in a row. LIMIT and PAUSE let you run fewer questions or slow down.

Why we do this: You need something to observe. This is a compact version of Lab 3: download the SEBI memo, chunk it, store it in ChromaDB, retrieve, and answer from the retrieved text only.

What the code does: retrieve_chunks returns the top matching chunks with page numbers. generate builds a prompt from them and asks the model, replying 'Not found in the provided documents.' when the context lacks the answer. It returns both the answer and the prompt, because the next exercise estimates tokens from the prompt.

Why we do this: When an answer is wrong or slow you need to know where: did retrieval fail, or did generation? A trace records one request from start to finish, and each step inside it is a span with its own timing and details (Module 8, section 2).

What the code does: The Trace class records spans. Its span() context manager times whatever runs inside the with block. finish() adds up time, estimated tokens and estimated cost. You complete rag_answer so it records a retrieve span and a generate span, storing token estimates in the generate span.

Why we do this: You cannot measure quality without a fixed set of questions with known right answers. Running the same set after every change shows whether you made things better or worse, which is a regression test.

What the code does: 20 questions written from the SEBI memo. 18 have answers in the document. 2 do not, because the memo excises the number of comments received and states no penalty, so the correct behaviour there is to say 'Not found'. Each item has keys: a list of groups, and an answer passes if at least one alternative from every group appears in it. You add two questions of your own after checking the memo text.

Why we do this: The fact check tells you whether key facts are present. It does not tell you whether the answer invented anything else. Groundedness asks whether every claim in the answer is supported by the retrieved text. An answer can be correct yet partly invented, and in finance that matters (Module 8, section 3).

What the code does: An LLM judge is a second model call that reads the sources and the answer and gives 1 (fully supported) or 0 (some unsupported claim). You write groundedness(context, answer): a judge prompt ending in SCORE: 1 or SCORE: 0, then a regular expression to parse it. evaluate runs every golden question and records correctness, groundedness, latency and cost.

Why we do this: Twenty rows are hard to act on. A production dashboard reduces them to a few numbers with thresholds, so a person or a CI job can say ship or do not ship (Module 8, section 10).

What the code does: You write summarise(rows) to compute correctness rate, groundedness rate, median and 95th-percentile latency, and cost per query. gates compares them with thresholds and prints PASS or FAIL for each. You then re-run with k=1 instead of 4, which simulates a cost-saving change someone might ship untested, and compare the two runs side by side.

Why we do this: Your Trace class teaches what a trace is. In practice you use a tool that stores traces, draws them as a timeline, and lets you search and compare them. Langfuse is an open-source option (Module 8, section 8).

What the code does: Reads your keys, then wraps the existing retrieve_chunks and generate functions with Langfuse's observe decorator. Because rag_answer calls them, each question appears as a trace with nested retrieve and generate steps. With no keys set, the cell skips itself.