{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Lab 5: Bias Audit and Model Card\n", "**Goal:** audit a credit-approval model for bias, explain it with SHAP, try one mitigation, and write the model card that documents all of it.\n", "\n", "**Why:** a model can be accurate overall and still treat groups unfairly, and in lending that is a legal and reputational risk as well as an ethical one. Regulators and model-risk teams expect evidence, not assurances. This is Module 5, sections 2 (model risk), 3 (explainability), 4 (bias and fairness) and 6 (audit). **No API key is needed.**\n", "\n", "**About the data:** the dataset is **synthetic**. You generate it inside the notebook with a fixed random seed, so everyone gets the same data. The generator deliberately builds in the kind of bias found in real lending history, so that you have something to find. Do not draw conclusions about real people or real lenders from it." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 0: Setup and the data\n", "**Why we do this:** we need a dataset where we *know* what is true. That lets us check whether the audit finds the problem we planted.\n", "\n", "**What the code does:** `make_data` creates 4,000 loan applicants with income, credit score, debt ratio, years employed, an `informal_work` flag and a `gender` column. Two outcomes are generated:\n", "- `repaid`: whether the person would really repay. It depends **only on financial features**.\n", "- `approved`: the *historical* decision. It follows the same financial features **plus a penalty for women and for informal workers**, mimicking a biased past process.\n", "\n", "Women are also more likely to be in informal work in this data. That makes `informal_work` a **proxy**: a feature that is not itself a protected attribute but is correlated with one.\n", "\n", "**Key terms:**\n", "- *Protected attribute:* a characteristic such as gender that decisions should not unfairly depend on.\n", "- *Proxy:* a feature that stands in for a protected attribute.\n", "- *Label bias:* the training answers (past approvals) are themselves unfair, so a model trained on them learns the unfairness.\n", "\n", "**Important:** using a binary gender column is a simplification. Real audits look at more groups and at intersections (for example gender and region together)." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "%pip install -q scikit-learn pandas numpy matplotlib shap" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "import numpy as np, pandas as pd\n", "import matplotlib.pyplot as plt\n", "from sklearn.ensemble import GradientBoostingClassifier\n", "from sklearn.model_selection import train_test_split\n", "\n", "def make_data(n=4000, seed=42):\n", " rng = np.random.default_rng(seed)\n", " female = rng.random(n) < 0.5\n", " income = rng.lognormal(mean=12.4, sigma=0.5, size=n)\n", " score = np.clip(rng.normal(680, 70, n), 300, 900)\n", " debt_ratio = np.clip(rng.beta(2, 5, n), 0, 1)\n", " years = np.clip(rng.gamma(3, 2.5, n), 0, 40)\n", " informal = rng.random(n) < np.where(female, 0.40, 0.12) # proxy: more common among women\n", " z_fin = 0.012 * (score - 650) + 0.9 * np.log(income / 240000) - 3.0 * (debt_ratio - 0.3) + 0.04 * years\n", " repaid = rng.random(n) < 1 / (1 + np.exp(-(z_fin + 1.2))) # truth: financial features only\n", " z_hist = z_fin + 0.6 - 0.8 * female - 1.2 * informal # history: biased against women and informal work\n", " approved = rng.random(n) < 1 / (1 + np.exp(-z_hist))\n", " return pd.DataFrame({\n", " \"gender\": np.where(female, \"F\", \"M\"), \"income\": income.round(0), \"credit_score\": score.round(0),\n", " \"debt_ratio\": debt_ratio.round(3), \"years_employed\": years.round(1), \"informal_work\": informal.astype(int),\n", " \"repaid\": repaid.astype(int), \"approved\": approved.astype(int)})\n", "\n", "df = make_data()\n", "print(df.head(), \"\\n\")\n", "print(df.groupby(\"gender\")[[\"approved\", \"repaid\"]].mean().round(3))\n", "print(\"\\ninformal work rate:\\n\", df.groupby(\"gender\")[\"informal_work\"].mean().round(3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** the average `repaid` rate is nearly the same for women and men (about 0.81 to 0.83), but the historical `approved` rate is clearly lower for women. Informal work is far more common among women.\n", "\n", "**What to notice:** equal real repayment but unequal approvals is the signature of a biased process. A model trained to copy the approvals will copy the bias too." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 1: Train a baseline model\n", "**Why we do this:** the audit needs a model to audit. We train a standard gradient-boosting classifier to predict the historical approval, which is exactly what a team would do if it trained on its past decisions.\n", "\n", "**What the code does:** splits the data 70/30 into training and test sets (stratified so both keep the same approval rate), builds a feature table, and trains two models: one that is allowed to use gender (`model_full`) and one that is not (`model_blind`). Dropping a protected column is called *fairness through unawareness*.\n", "\n", "**Key terms:** *Test set* = data the model never saw during training, used for honest measurement.\n", "\n", "**Finance link:** many lenders do not feed gender to credit models. Whether that is enough is exactly what this lab tests. Rules on which features may be used differ by country and product, so check with your compliance team before making any real decision." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "X = df.drop(columns=[\"repaid\", \"approved\", \"gender\"]).copy()\n", "X[\"gender_F\"] = (df[\"gender\"] == \"F\").astype(int)\n", "y = df[\"approved\"]\n", "\n", "X_train, X_test, y_train, y_test, g_train, g_test, r_train, r_test = train_test_split(\n", " X, y, df[\"gender\"], df[\"repaid\"], test_size=0.3, random_state=0, stratify=y)\n", "\n", "BLIND = [c for c in X.columns if c != \"gender_F\"]\n", "model_full = GradientBoostingClassifier(random_state=0).fit(X_train, y_train)\n", "model_blind = GradientBoostingClassifier(random_state=0).fit(X_train[BLIND], y_train)\n", "\n", "pred_full = model_full.predict(X_test)\n", "pred_blind = model_blind.predict(X_test[BLIND])\n", "print(\"accuracy, uses gender:\", round((pred_full == y_test).mean(), 3))\n", "print(\"accuracy, gender removed:\", round((pred_blind == y_test).mean(), 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** two accuracy figures around 0.73 to 0.76, close to each other.\n", "\n", "**What to notice:** accuracy alone says both models are fine. Accuracy measures how well the model copies the old decisions, including the unfair ones. That is why the next step looks at fairness separately." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 2: Measure fairness\n", "**Why we do this:** fairness has to be *measured*, and different definitions answer different questions (Module 5, section 4).\n", "\n", "**The three measures:**\n", "- *Selection rate:* the share of a group that the model approves.\n", "- *Disparate impact ratio (the 80% rule):* the protected group's selection rate divided by the reference group's. A value below 0.8 is the common red flag.\n", "- *Equal opportunity gap:* among people who really would repay (`repaid == 1`), how much more often does the model approve the reference group than the protected group? This is a true-positive-rate difference, and it uses the real outcome, not the biased history.\n", "\n", "**Your task:** complete the three functions. `pred` is a NumPy array of 0/1 predictions, `group` an array of \"F\"/\"M\", and `repaid` an array of 0/1.\n", "\n", "**Hints:** a selection rate is `pred[group == g].mean()`. The true positive rate is the same mean restricted to `(group == g) & (repaid == 1)`." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def selection_rate(pred, group, g):\n", " # YOUR CODE HERE: share of group `g` that is predicted 1\n", " raise NotImplementedError\n", "\n", "def disparate_impact(pred, group, protected=\"F\", reference=\"M\"):\n", " # YOUR CODE HERE: selection rate of `protected` divided by selection rate of `reference`\n", " raise NotImplementedError\n", "\n", "def equal_opportunity_gap(pred, group, repaid, protected=\"F\", reference=\"M\"):\n", " # YOUR CODE HERE: TPR(reference) - TPR(protected), where TPR(g) = mean(pred) over group g AND repaid == 1\n", " raise NotImplementedError\n", "\n", "def audit(pred, label):\n", " g, rep = g_test.values, r_test.values\n", " print(f\"{label:22} SR_F={selection_rate(pred, g, 'F'):.3f} SR_M={selection_rate(pred, g, 'M'):.3f} \"\n", " f\"DI={disparate_impact(pred, g):.3f} TPR gap={equal_opportunity_gap(pred, g, rep):.3f} \"\n", " f\"{'<-- below 0.8' if disparate_impact(pred, g) < 0.8 else ''}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def selection_rate(pred, group, g):\n", " return pred[group == g].mean()\n", "\n", "def disparate_impact(pred, group, protected=\"F\", reference=\"M\"):\n", " return selection_rate(pred, group, protected) / selection_rate(pred, group, reference)\n", "\n", "def equal_opportunity_gap(pred, group, repaid, protected=\"F\", reference=\"M\"):\n", " tpr = lambda g: pred[(group == g) & (repaid == 1)].mean()\n", " return tpr(reference) - tpr(protected)\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "audit(pred_full, \"model uses gender\")\n", "audit(pred_blind, \"gender removed\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** for the model that uses gender, the disparate impact ratio is below 0.8 (around 0.7) and the equal-opportunity gap is large (around 0.2). With gender removed both improve but **do not disappear**: the gap stays clearly above zero.\n", "\n", "**What to notice:** removing the protected column helped, but did not fix the problem, because `informal_work` carries part of the same signal. This is the proxy problem. The equal-opportunity gap says something worse than the selection rates alone: women who would repay are approved less often.\n", "\n", "**Finance link:** a ratio below 0.8 is not automatically unlawful, and a ratio above 0.8 is not automatically safe. These tests start an investigation. Which measure applies depends on the law and the product (Module 5, section 4)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 3: Explain the model with SHAP\n", "**Why we do this:** to see *why* the model behaves this way. SHAP assigns each feature a contribution to each individual prediction (Module 5, section 3). Averaging the absolute contributions gives a global importance ranking.\n", "\n", "**What the code does:** computes SHAP values for both models on the test set and ranks features by mean absolute contribution. If the `shap` package fails to install, the cell falls back to permutation importance, a simpler method that measures how much accuracy drops when a feature is shuffled.\n", "\n", "**Key terms:**\n", "- *SHAP value:* how much one feature pushed one prediction up or down.\n", "- *Global importance:* the average size of those pushes across many predictions.\n", "\n", "**Your task:** complete `rank_of(importance, feature)`, returning the 1-based position of `feature` in the importance ranking. Hint: `list(importance.index).index(feature) + 1`." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def importance_table(model, X_eval):\n", " try:\n", " import shap\n", " sv = shap.TreeExplainer(model).shap_values(X_eval)\n", " if isinstance(sv, list):\n", " sv = sv[-1]\n", " return pd.Series(np.abs(sv).mean(axis=0), index=X_eval.columns).sort_values(ascending=False), \"SHAP\"\n", " except Exception as e:\n", " from sklearn.inspection import permutation_importance\n", " r = permutation_importance(model, X_eval, y_test, n_repeats=5, random_state=0)\n", " return pd.Series(r.importances_mean, index=X_eval.columns).sort_values(ascending=False), f\"permutation (SHAP unavailable: {type(e).__name__})\"\n", "\n", "imp_full, how = importance_table(model_full, X_test)\n", "imp_blind, _ = importance_table(model_blind, X_test[BLIND])\n", "print(\"method:\", how)\n", "print(\"\\nmodel that uses gender:\\n\", imp_full.round(3))\n", "print(\"\\nmodel without gender:\\n\", imp_blind.round(3))\n", "\n", "def rank_of(importance, feature):\n", " # YOUR CODE HERE: 1-based rank of `feature` in the importance ranking\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def rank_of(importance, feature):\n", " return list(importance.index).index(feature) + 1\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "print(\"gender_F rank (model with gender):\", rank_of(imp_full, \"gender_F\"), \"of\", len(imp_full))\n", "print(\"informal_work rank (model with gender):\", rank_of(imp_full, \"informal_work\"), \"of\", len(imp_full))\n", "print(\"informal_work rank (model without gender):\", rank_of(imp_blind, \"informal_work\"), \"of\", len(imp_blind))\n", "imp_blind.sort_values().plot.barh(title=\"Feature importance, model without gender\"); plt.tight_layout(); plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** credit score ranks first. `informal_work` ranks surprisingly high, above income in our run, and `gender_F` has a smaller but non-zero effect in the first model. In the model without gender, `informal_work` moves up and carries more of the load. Exact values depend on the method (SHAP or permutation) and library versions.\n", "\n", "**What to notice:** that is the proxy at work. The model has found another route to the same information.\n", "\n", "**Important:** SHAP explains what the *model* does, not what is *fair*. A feature can be important and perfectly legitimate, or important because it carries bias. Judging which is a human decision, informed by the fairness measures above.\n", "\n", "**Finance link:** SHAP-style reason codes are one way to support adverse-action explanations to applicants (Module 5, section 3)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 4: Try a mitigation: reweighing\n", "**Why we do this:** an audit that finds a problem should lead to something being done. We try one *pre-processing* method from Module 5, section 4.\n", "\n", "**What the code does:** *Reweighing* gives each training example a weight so that, in the weighted data, group membership and the approval label are statistically independent. Under-approved group-and-label combinations are weighted up and over-represented ones down. The model is then retrained on the weighted data.\n", "\n", "**The weight formula:** for a group *g* and label *y*, `weight = P(g) \u00d7 P(y) / P(g and y)`, where each probability is the share of the training data.\n", "\n", "**Your task:** complete `reweighing_weights(group, y)`. It returns a NumPy array with one weight per training row. Hint: loop over each group and each label value, build a boolean mask, and set the weights for those rows with the formula.\n", "\n", "**Important:** reweighing improves a statistical measure on the training process. It does not make a decision fair by itself, and it can cost accuracy. Treating groups differently when making an individual decision can itself be restricted by law in some places, so involve your legal and compliance teams before choosing a method for real use." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def reweighing_weights(group, y):\n", " # YOUR CODE HERE: return weights = P(g) * P(y) / P(g and y) for every row\n", " raise NotImplementedError" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
Solution (try first)\n", "\n", "```python\n", "def reweighing_weights(group, y):\n", " group, y = np.asarray(group), np.asarray(y)\n", " w = np.ones(len(y))\n", " for g in np.unique(group):\n", " for label in np.unique(y):\n", " mask = (group == g) & (y == label)\n", " w[mask] = ((group == g).mean() * (y == label).mean()) / mask.mean()\n", " return w\n", "```\n", "
" ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "w = reweighing_weights(g_train.values, y_train.values)\n", "model_rw = GradientBoostingClassifier(random_state=0).fit(X_train[BLIND], y_train, sample_weight=w)\n", "pred_rw = model_rw.predict(X_test[BLIND])\n", "\n", "audit(pred_full, \"uses gender\")\n", "audit(pred_blind, \"gender removed\")\n", "audit(pred_rw, \"removed + reweighed\")\n", "print(\"\\naccuracy: uses gender\", round((pred_full == y_test).mean(), 3),\n", " \"| removed\", round((pred_blind == y_test).mean(), 3), \"| reweighed\", round((pred_rw == y_test).mean(), 3))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** a three-row comparison. Reweighing usually moves the disparate impact ratio closer to 1 and shrinks the equal-opportunity gap further, at a small cost in accuracy. Exact numbers depend on the library versions.\n", "\n", "**What to notice:** the improvement is real but partial. The gap is smaller, not zero. That honest result is what goes in the model card, along with the trade-off. Other methods exist (in-processing and post-processing, Module 5, section 4), and none of them removes the need for human judgement." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Exercise 5: Write the model card\n", "**Why we do this:** a *model card* is a short, standard document that records what a model is for, how it was built, how it performs, and where it should not be used. Model-risk reviewers and auditors ask for it (Module 5, sections 2 and 6, documentation requirements).\n", "\n", "**What the code does:** `build_card` fills a template with the numbers you just computed, so the card cannot drift out of date with the results. You write the two sections that need human judgement: the limitations and the human-oversight plan.\n", "\n", "**Your task:** write the strings `limitations` and `oversight` in the cell below. Be specific and honest. Mention what the audit found, what it did not cover, and who reviews decisions.\n", "\n", "**Important:** the card describes this *synthetic* exercise model. In real work a card is written by the model owner, reviewed independently, and updated with every retrain." ] }, { "cell_type": "code", "metadata": {}, "execution_count": null, "outputs": [], "source": [ "def card_numbers(pred):\n", " g, rep = g_test.values, r_test.values\n", " return {\"acc\": (pred == y_test).mean(), \"di\": disparate_impact(pred, g), \"gap\": equal_opportunity_gap(pred, g, rep)}\n", "\n", "before, after = card_numbers(pred_blind), card_numbers(pred_rw)\n", "\n", "limitations = \"\" # YOUR CODE HERE: 3-5 specific limitations (data is synthetic, binary gender only, one metric set, ...)\n", "oversight = \"\" # YOUR CODE HERE: who reviews declined applications, how often the audit re-runs, what triggers escalation\n", "\n", "def build_card():\n", " return f'''# Model Card: Credit Approval Model (synthetic training exercise)\n", "\n", "## Intended use\n", "Teaching example of a bias audit. Not for real lending decisions.\n", "\n", "## Data\n", "Synthetic dataset of 4,000 applicants (seed 42) with deliberately injected historical bias against women and informal workers.\n", "Features: income, credit score, debt ratio, years employed, informal-work flag. Label: historical approval.\n", "\n", "## Model\n", "Gradient-boosting classifier. Variants audited: with gender, gender removed, gender removed plus reweighing.\n", "\n", "## Performance (test set)\n", "- Accuracy, gender removed: {before['acc']:.3f}\n", "- Accuracy, gender removed + reweighed: {after['acc']:.3f}\n", "\n", "## Fairness results (protected group F vs reference group M)\n", "| Measure | Gender removed | Removed + reweighed |\n", "|---|---|---|\n", "| Disparate impact ratio (target >= 0.8) | {before['di']:.3f} | {after['di']:.3f} |\n", "| Equal opportunity gap (target near 0) | {before['gap']:.3f} | {after['gap']:.3f} |\n", "\n", "## Explainability\n", "SHAP (or permutation importance) rankings were reviewed. The informal-work flag acts as a proxy for gender.\n", "\n", "## Limitations\n", "{limitations}\n", "\n", "## Human oversight\n", "{oversight}\n", "'''\n", "\n", "card = build_card()\n", "print(card)\n", "open(\"model_card.md\", \"w\", encoding=\"utf-8\").write(card)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Expected output:** the finished card printed, with your two sections filled in, and a file `model_card.md` saved next to the notebook.\n", "\n", "**What to notice:** every number in the fairness table came from code you wrote, so a reviewer can re-run it. Notice also what the card has to admit: a residual gap remains, the data is synthetic, and one binary attribute is a simplification. A card that hides these is worse than no card.\n", "\n", "**Finance link:** this is the document a model-validation team or regulator asks for first, and the audit trail behind it (Module 5, sections 2 and 6).\n", "\n", "## Stretch goals\n", "1. Add a second protected attribute (for example region) to the generator and check intersectional groups.\n", "2. Plot approval rate by credit-score band for each gender and look for where the gap is widest.\n", "3. Write a model card for the Lab 4 research agent instead: intended use, tools, evaluation results, and escalation rules.\n", "4. Try post-processing: choose different decision thresholds per group, then discuss whether that is acceptable in your jurisdiction.\n", "\n", "## Common errors\n", "- **`NotImplementedError`:** you have not completed that function yet.\n", "- **`shap` fails to install or run:** the notebook falls back to permutation importance automatically.\n", "- **Numbers differ slightly from the expected ranges:** library versions can change model internals. The patterns (a gap that shrinks but does not vanish) should still hold.\n", "- **Empty limitations or oversight in the card:** you left the placeholders blank. Fill in both strings." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }