AI Observability, Evaluation & Tracing

The end-to-end journey of one request through your AI pipeline. A trace contains spans — each span is one step (embedding query, vector search, LLM call, tool call). Traces let you see the full picture: what went into the model, what came out, which step was slow.

Quantitative measurements aggregated over time. Unlike traces (per-request), metrics tell you about trends: is accuracy declining? Is cost per query rising? Are injection attempts increasing?

Structured records of events — prompts, responses, tool arguments, errors. More detail than metrics, more queryable than traces. In finance, logs are the audit trail. They must be stored securely, with PII redacted, and retained per policy.

Run against a fixed dataset before any change goes to production. Fast, cheap, reproducible. Your first gate before a prompt change or model upgrade ships.

500 financial Q&A pairs with known correct answers. Run nightly. Fail the CI build if accuracy drops >2%.

Sample live traffic and score it in real time (or near real time). Catches drift that offline evals miss — real user queries are always messier than your test set.

Score 5% of live compliance queries for groundedness. Alert if 7-day rolling average drops below 0.85.

Domain experts review a sample of outputs and label them. The ground truth — but slow and expensive. Use for calibrating automated metrics and for high-stakes decisions.

Weekly: compliance officers review 20 flagged outputs. Their labels feed back into the golden dataset.

Use a separate, more capable model to evaluate outputs at scale. Much faster and cheaper than human eval. Can be biased — always calibrate against human labels before trusting it.

Use Claude Opus to score whether a credit analysis output correctly identifies all material risks. Calibrated on 200 human-labelled examples first.

Azure AI Foundry — Built-in Evaluators

AI Foundry (formerly Azure AI Studio) includes built-in evaluators you run directly on your data, no code needed.

• Groundedness — is the response supported by the provided context?
• Coherence — is the response logically structured and clear?
• Fluency — is the language natural and well-formed?
• Relevance — does the response address the query?
• Similarity — how close to a reference answer?
• Safety evaluators — violence, self-harm, sexual content, hate — all scored by severity

Azure Monitor + Application Insights

Standard Azure monitoring extended for LLM workloads. Captures custom metrics, traces, and logs from your AI gateway.

• Custom metrics: emit token_count, cost, quality_score as App Insights custom events
• Distributed tracing: correlate LLM spans with upstream HTTP requests via X-Correlation-ID
• Alert rules: fire when hallucination_rate > 0.1 or latency_p95 > 5s
• Workbooks: pre-built dashboards for LLM cost and quality trends
• Log Analytics: KQL queries over all prompt/response logs (PII redacted)

Azure AI Content Safety Monitoring

Real-time monitoring of content safety decisions across all apps using your gateway.

• Prompt Shield hit rate — how often injection attacks are detected
• Harm category distribution — which categories are being triggered
• False positive tracking — which legitimate queries are being blocked
• Trend alerts — spike in injection attempts may indicate active attack

Microsoft Purview — Data Lineage for AI

Tracks where data flows through your AI system — essential for compliance and right-to-be-forgotten.

• Sensitivity label propagation — if source doc is Confidential, the RAG chunk is Confidential
• Data lineage: trace which source documents fed which model responses
• DLP integration — alert when data classified as MNPI reaches an unauthorised AI app
• Audit reports for regulators: where did this client's data go?

Enable per-model-deployment. Logs every request/response to S3 or CloudWatch Logs. Configurable — log full prompts, or metadata only (for PII-sensitive workloads). Queryable via Athena for cost analysis.

Custom metrics from Bedrock: InvocationLatency, InputTokenCount, OutputTokenCount. Build dashboards and alarms. Cost alert: alarm when daily_tokens > budget. Latency alert: alarm when p95 latency > 4s.

Trace requests across Lambda → Bedrock → DynamoDB → S3 in one trace. See the full request path including the LLM call. Identify which part of your agent pipeline is slow or failing.

Bias and explainability monitoring for ML models (including fine-tuned models deployed on SageMaker). Monitors for data drift and model drift over time. More relevant for fine-tuned/traditional ML models than Bedrock-hosted LLMs.

Bedrock Guardrails (AWS's content filtering layer) provides built-in metrics: how many requests triggered PII filters, topic restrictions, grounding checks. Dashboard in AWS console, metrics in CloudWatch.

Centralise Bedrock invocation logs, CloudTrail API calls, and GuardDuty findings into Security Lake (OCSF format). Full audit trail for compliance. Query with Athena. Forward to third-party SIEM if needed.

Built-in evaluators for Gemini models and custom models: summarisation quality, QA accuracy, grounding, text generation quality. Run against datasets stored in BigQuery. Results stored in BigQuery for trend analysis.

Distributed tracing for all GCP services. Capture LLM call latency, token counts as custom attributes on spans. Integrate with OpenTelemetry for vendor-neutral instrumentation.

Emit custom metrics (token counts, quality scores, cost) from your Vertex AI apps. Build dashboards. Set alerting policies. Cloud Logging stores all LLM request/response logs — query with log-based metrics.

Monitor deployed models for training-serving skew and prediction drift. Most relevant for fine-tuned models and traditional ML models deployed on Vertex. Gemini API monitoring is handled separately via Cloud Monitoring.

Store all eval results, trace data, and quality metrics in BigQuery. Run SQL analysis over months of eval history. Create Looker dashboards for business-facing quality reporting. Cost-effective at scale.

All Vertex AI API calls are captured in Admin Activity and Data Access audit logs. Essential for compliance: who called which model, with what data, when. Forward to Chronicle SIEM for security analysis.

PII in Logs — Non-Negotiable

Every prompt and response that goes through your AI system is a potential log. Logs are stored, indexed, and accessed by operations teams, vendors, and regulators. PII must be redacted before logging.

# Redact before logging — use Presidio
from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

def redact_for_logging(text: str) -> str:
    results = analyzer.analyze(text, language="en")
    return anonymizer.anonymize(
        text=text,
        analyzer_results=results,
    ).text

# In your gateway — redact prompt BEFORE logging
def audit_log(prompt, response, metadata):
    log_entry = {
        "prompt": redact_for_logging(prompt),
        "response": redact_for_logging(response),
        "metadata": metadata,
        "timestamp": datetime.utcnow().isoformat(),
    }
    siem_client.send(log_entry)