Monitoring AI Systems at Scale

Costs 10x overnight from a single prompt change. A RAG pipeline silently serves irrelevant context for three weeks. An agent loops on tool calls until the timeout kills it, but the p95 latency chart looks fine. These aren’t hypotheticals — we’ve watched each one happen in production, and traditional monitoring caught exactly none of them.

When we started serving AI requests at scale — millions per day across dozens of models and pipelines — we brought our standard SRE playbook. Latency dashboards. Error rate alerts. Throughput tracking. And we learned the hard way that monitoring AI systems requires an entirely different set of signals. This article covers what we measure, how we measure it, and the alerting patterns that separate signal from noise when your system’s output is fundamentally non-deterministic.

The Four Pillars Aren’t Enough

The canonical four golden signals of monitoring — latency, traffic, errors, and saturation — were designed for systems where correct behavior is well-defined. A request either returns a 200, a 500, or times out. A database either has room on the disk or it doesn’t. These signals are necessary for AI systems too, but they are nowhere near sufficient.

An AI service can return a 200 with a perfect latency percentile and still be completely broken. The model might be emitting the same three sentences for every input. The embedding service could have swapped to a quantization method that cut recall by 40%. The prompt template might have a formatting bug that causes the model to ignore the instructions entirely. None of these show up in your error rate.

We group our AI-specific monitoring into five categories that layer on top of traditional observability:

We’ll walk through each of these in detail.

Token Usage and Cost Observability

The single most expensive incident we ever had was invisible to every standard monitoring tool. A team changed a system prompt from “Answer concisely” to “Provide a thorough response.” The output length went from ~150 tokens per request to ~1,200. Our daily LLM cost increased from $400 to $4,200 in under 24 hours.

No alert fired. The latency chart showed a modest uptick, but still within the p99 SLO. The error rate was flat. The throughput was unchanged. The finance team noticed the bill two weeks later.

Here’s what we track for every single request:

• Input tokens — prompt, system message, chat history, tool definitions, few-shot examples. Each component tracked separately so we can spot prompt bloat.
• Output tokens — completion length, and whether it hit the max_tokens ceiling (a hard stop there often indicates a truncated response).
• Cached tokens — prompt caching hits and misses, with the dollar value saved per day.
• Cost per request — computed in real-time using the model’s per-token pricing, broken into input and output cost.

On our cost dashboard, we track:

Cost per model per day with a stacked area chart. The moment a new model deployment starts costing more than the previous one, we see it within 15 minutes, not 15 days.

Cost per request by model as a heatmap over time. A single cell going from $0.003 to $0.03 is a prompt change, not a traffic shift.

Cost per pipeline stage. In a RAG pipeline, the embedding step might cost $0.0001 per request while the generation step costs $0.02. If those ratios shift, something changed in how the pipeline is configured.

Cost per user or tenant. When one customer’s usage pattern shifts and their per-request cost triples, we catch it before it becomes a billing escalation.

The implementation is straightforward: emit a metric with tags for model, pipeline, deployment_version, and request_id from the serving layer. Aggregate into a time-series database. Alert on per-request cost changes beyond 2x of the 7-day rolling average.

Latency by Model and by Pipeline Stage

Standard request latency — total wall time from ingress to response — is useful but hides where time is actually spent. An AI pipeline has distinct stages with very different performance characteristics:

We instrument every stage with a span and emit stage-level latency metrics tagged by model name and pipeline ID. This catches problems like:

A vector database index falling behind. The retrieval stage p50 went from 80ms to 340ms over a week because the index auto-rebuild triggered on a growing dataset. The total request latency stayed within bounds — the LLM generation time dominated — so standard monitoring saw nothing. The stage-level chart showed the retrieval stage gradually degrading.

GPU contention from a noisy neighbor. When multiple model deployments share the same GPU, generation latency for one model can spike when another model gets a burst of traffic. Stage-level latency by deployment instantly reveals the correlation.

Prompt template changes that increase token count. A new prompt version that adds 500 tokens of instructions doesn’t just increase cost — it increases generation latency proportionally. We track latency per thousand output tokens as a normalized metric. If that normalized number changes, the model or hardware changed, not just the prompt.

Response Quality Proxies

Measuring actual output quality in production is hard. Human evaluation doesn’t scale, and LLM-as-judge evaluation adds latency and cost. We don’t try to measure quality directly on every request. Instead, we track proxies that correlate strongly with problems:

Refusal Rate

When a model starts refusing requests it shouldn’t — or stops refusing requests it should — that’s a critical signal. We track the percentage of responses that start with “I cannot,” “I’m sorry,” “I’m not able to,” and similar patterns. A sudden drop in refusal rate might mean the guardrails are broken. A sudden spike might mean the safety classifier changed behavior.

Our alert: refusal rate deviates more than 3 standard deviations from its 7-day rolling baseline. These alerts have caught two real incidents: a content filter accidentally set to “strict” mode (refusal rate went from 2% to 18%), and a model update that silently removed safety fine-tuning (refusal rate dropped from 2% to 0.1%).

Response Length as a Quality Proxy

We track output token count per request bucketed by input category. When responses get shorter for no apparent reason, the model might be truncating, the prompt might have a conflict, or the sampling parameters might have changed. When they get longer, cost and latency go up.

Citation and Source Quality

For RAG-based systems, we track:

• Citation rate: What percentage of responses include citations? A drop might mean the retriever is failing silently.
• Source diversity: How many different documents are being cited? If every response cites the same three documents, the retrieval is collapsing to a local neighborhood.
• Citation-to-claim ratio: Are responses making unsupported claims? Rough heuristic: responses that make claims without citations often correlate with hallucinations.

Repetition and Degeneration Flags

A model stuck in a loop — repeating the same phrase or structure — produces no error code but terrible output. We check for n-gram repetition at the token and sentence level. If the repetition rate exceeds a threshold (usually 5x the baseline), we flag the response. At our scale, about 0.3% of responses show pathological repetition patterns. Catching them means we can block those responses from reaching users.

Embedding and Output Distribution Drift

This is the monitoring category that catches the most silent degradations. Embedding drift and output distribution drift happen gradually, over days or weeks. They are invisible to point-in-time metrics. But they are among the most destructive failure modes for AI systems.

Embedding Drift

Your embedding model produces a vector for every input. The distribution of those vectors across all requests forms a shape — the centroid, the variance, the density clusters. When that distribution shifts, it means the inputs your system is receiving have changed. This matters because:

• The retriever in your RAG pipeline was tuned on the old distribution. Retrieval quality degrades.
• The prompt templates were designed for the old patterns. The model may misinterpret the new ones.
• The content filters and safety classifiers were calibrated on the old distribution. Their accuracy drifts.

We track embedding drift using two metrics:

Mean cosine similarity to a reference centroid. We compute the centroid of embeddings from a reference period (typically the last 30 days). For each batch of new requests, we compute the mean cosine similarity between new embeddings and the reference centroid. A downward trend means the input distribution is drifting.

Out-of-distribution fraction. We maintain a simple density estimate of the embedding space. For each new embedding, we check whether it falls in a low-density region. The fraction of OOD requests is a leading indicator of problems. When it exceeds 5%, we trigger a review.

Output Distribution Shift

The statistical properties of your model’s outputs change over time. We track:

• Output length distribution. Is the model getting more verbose or more terse?
• Token frequency distribution. Are certain tokens or phrases appearing more or less often?
• Sentiment distribution. For customer-facing applications, a shift toward negative language is a red flag.
• Category or label distribution. For classification tasks, are the relative frequencies of output labels stable?

A concrete example: We had a model that handled customer intent classification. The “escalate to human” category went from 12% of responses to 34% over two weeks. The model quality hadn’t changed — the user base had. A product update had brought in a different demographic. The output distribution shift told us before the support team started reporting the increase.

Prompt Template Version Tracking

Every prompt template in production is versioned. We tag every request with the template version and track:

• Which version is serving what percentage of traffic.
• The performance characteristics of each version (latency, cost, refusal rate, response length).
• Whether a new version has statistically different behavior from the previous one.

We deploy new prompt versions to 5% of traffic first and compare metrics against the control group for at least 24 hours. Without this, you are flying blind on every prompt edit.

Trace-Based Monitoring for Multi-Step Workflows

Aggregated metrics are smoothed. They hide the pathological patterns that only appear at the individual trace level. For AI systems — especially those with multi-step pipelines — trace-based monitoring is not optional.

What We Trace

Every request gets a distributed trace that spans:

• The entire request lifecycle, from ingress to response.
• Every internal step: embedding generation, retrieval, prompt assembly, LLM call, response parsing, content filtering, post-processing.
• Sub-calls within steps: if the LLM makes tool calls, each tool invocation is a child span.
• External dependencies: vector database queries, external API calls, cache lookups.

Tool Call Loops

This is the biggest blind spot in aggregate metrics. An agent makes a tool call, gets a result, decides to make another tool call, and another, and another. Each call succeeds. The latency per call is reasonable. But the total request takes 45 seconds and calls the same tool 18 times in a row.

We identify tool call loops by analyzing trace structure:

• Call depth. How deep is the tool call stack? A normal RAG pipeline might have depth 2-3 (retrieve, generate). An agent in trouble might have depth 15+.
• Repeat tool calls. Is the same tool being invoked with similar parameters? We flag traces where the same tool is called more than 5 times.
• Cancellation and timeout rates per step. Which pipeline stage is most likely to be cancelled or timed out? That’s your bottleneck.

Context Window Utilization

For every LLM call, we track:

• Input tokens vs. max context window. Are you hitting context limits? Are requests being truncated?
• Retrieved context quality. In RAG pipelines, how many of the retrieved chunks actually fit in the context window, and how many are truncated?
• Context wastage. Are you sending 8,000 tokens of chat history when only 500 tokens matter?

Tracking this at the trace level reveals systemic issues. We once found that a prompt template was accidentally including the full conversation history for every sub-request in a chain-of-thought pipeline. Each sub-request was sending 15,000 tokens of history for a task that needed 200. The trace data made the pattern obvious within minutes.

Setting SLOs for Non-Deterministic Systems

Traditional SLOs assume deterministic behavior. A request either succeeds or fails within a time threshold. AI systems produce outputs that vary even with the same input. What does “reliable” mean in this context?

We define three tiers of SLOs:

Availability SLOs (Deterministic)

These are the standard ones: the service must respond, and it must respond within a time budget. We set:

• p99 latency < 5 seconds for simple completions.
• p99 latency < 15 seconds for multi-step agentic workflows.
• Error rate < 0.1% for HTTP 5xx responses, timeouts, and rate limit hits.
• Uptime > 99.9% for the API endpoint.

These are non-negotiable. If the service doesn’t respond, nothing else matters.

Quality SLOs (Non-Deterministic)

These are the ones that require judgment:

• Refusal rate between 1% and 5%. Below 1% suggests guardrails are too weak. Above 5% suggests they’re too aggressive.
• Citation rate > 80% for RAG responses.
• Response length within 50% of expected range. A response that’s 20 tokens when the expected range is 200-400 is broken.
• Repetition rate < 0.5% of responses showing pathological patterns.

We measure these as service-level indicators (SLIs) with error budgets. If refusal rate stays above 5% for 24 hours, that’s budget consumption. When the budget is exhausted, we have a formal review.

Cost SLOs (Business)

• Per-request cost deviation < 2x of 7-day rolling average. If a request type starts costing more than double its normal cost, we need to know.
• Daily total cost within forecast ± 20%. Our forecasting model predicts daily cost based on traffic patterns and model mix. Exceeding the forecast by 20% triggers an investigation.
• Cost per pipeline step within expected ratios. If the embedding step suddenly accounts for 30% of total cost instead of its usual 5%, something changed.

Alerting Patterns That Work

Alert fatigue kills incident response. If every alert is important, no alert is important. For AI systems, the challenge is that many failure modes look like noise — a few bad responses, a modest drift, a temporary cost spike.

We follow three alerting principles:

Principle 1: Alert on Error Budget Consumption, Not Every Violation

A single p99 latency spike doesn’t need a page. It needs a graph. We page only when the error budget burn rate exceeds a threshold — currently set to 5% of the monthly budget consumed in 1 hour, or 10% in 24 hours. Everything else goes to a dashboard or a daily digest.

Principle 2: Use Statistical Baselines, Not Fixed Thresholds

Fixed thresholds are a trap. A 2% refusal rate might be fine for one model and catastrophic for another. We use rolling window baselines — typically 7 days — and alert on deviations that exceed 3 standard deviations from the baseline. This adapts automatically to changes in traffic patterns, user base, and model behavior.

Principle 3: Downgrade Alerts That Fire During Maintenance

We tag all deployments with version and environment. Alerts on a deployment that is actively being rolled out — version changed within the last 30 minutes, or traffic percentage is still ramping — are automatically downgraded from page to ticket. This prevents the 3 AM page because someone deployed a new model variant that hasn’t finished rolling out.

Our Alert Tiers

The Monitoring Stack: What to Use at Each Stage

The right tooling depends on your scale. Here’s what we’ve seen work:

Small Scale (Up to 100K requests/day)

You don’t need a custom observability platform. You need structured logging with rich tags and a decent analytics tool.

• Log aggregation: Something cheap that accepts JSON logs with tags. At this scale, you can parse structured logs to extract all the metrics we’ve described.
• Metrics: A simple time-series store with tagging support. Track cost, latency by stage, refusal rate, and response length distribution.
• Traces: Not essential at this scale unless you’re running multi-step agents. If you are, a minimal trace store that handles a few hundred spans per second is enough.
• Dashboards: A few focused dashboards — cost, latency by stage, quality proxies, drift indicators. Don’t build a dashboard for everything. Build one for each category.

Medium Scale (100K to 10M requests/day)

You need dedicated observability infrastructure:

• Traces become mandatory. You cannot debug multi-step workflows at this scale without trace data. Budget for a trace store that handles 1,000+ spans per second.
• Metrics need dimensionality. Tag-based metric systems become essential for slicing by model, pipeline, deployment version, and tenant.
• Cost tracking needs to be real-time. You cannot wait for the monthly bill. Per-request cost computation should be part of your serving infrastructure.
• Alerting needs baselines. Static thresholds will produce too many false positives. Invest in anomaly detection or at least rolling-window statistical baselines.

Large Scale (10M+ requests/day)

At this scale, you’re building custom tooling:

• Custom cost attribution engine. Per-request cost is table stakes. You also need per-tenant, per-feature, per-team cost allocation.
• Quality monitoring pipeline. A low-latency streaming evaluation pipeline that runs quality checks on every response, sampling 100% of traffic for critical proxies and a percentage for deeper evaluation.
• Drift detection in production. Automated comparison of embedding distributions and output distributions against reference baselines, with alerts on statistically significant shifts.
• Automated rollback triggers. When certain SLIs breach thresholds, the system can automatically roll back the deployment or shift traffic to a fallback model.

Our Stack

We use a combination of open-source and managed tools:

• Traces: OpenTelemetry-based instrumentation feeding into a distributed trace store. Every request gets a trace ID that spans all pipeline stages.
• Metrics: A dimensional time-series database with custom aggregations for token counts, cost, and stage latency. Alerting rules use windowed baselines.
• Logs: Structured JSON logging with consistent tag schemas across all services. Each log line includes model, pipeline, deployment version, trace ID, and tenant ID.
• Dashboards: A layered approach — high-level overview screens for on-call, detail screens for deep dives, and weekly reports for trend analysis.

Real Incidents Caught by Good Monitoring

To make this concrete, here are three incidents that our AI-specific monitoring caught, each of which would have been invisible to traditional monitoring:

The Prompt That Cost $3,800 Extra Per Day

A new prompt template was deployed to 10% of traffic. Our cost-per-request metric showed that requests using the new template cost $0.042 vs. $0.007 for the old template — a 6x increase. The cause: a system message that included the full knowledge base index as context. The fix took 12 minutes. Without per-template cost tracking, this would have rolled to 100% and cost $38,000 before the next billing cycle.

The Embedding Model Swap That Broke Retrieval

A vendor updated their embedding model. The new version had higher accuracy benchmarks. What they didn’t mention was that the embedding dimension changed and the distribution shifted significantly. Our embedding drift detector flagged a 40% drop in cosine similarity to the reference centroid within two hours of the deployment. The retrieval quality dashboard showed citation rates dropping from 85% to 62%. We rolled back and worked with the vendor on a migration plan, all before any user reported degraded results.

The Agent Loop That Wouldn’t Die

A multi-step agent started calling the same search tool repeatedly — 14 times per request on average, up from the normal 2-3. Each call succeeded and returned results. The p99 latency stayed under the SLO because the tool responded quickly. But the cost per request increased 5x. Our tool call depth metric flagged the anomaly. The trace explorer showed the pattern immediately: the agent was getting partial results and re-querying instead of synthesizing what it had. The fix was a prompt update that clarified when to stop searching and start answering.

Practical Takeaways

1. Track cost per request in real-time. This is the single highest-ROI monitoring investment you can make. A prompt change that increases output tokens by 500 costs real money, and you won’t see it on the billing dashboard for weeks. Compute cost on every request and aggregate by model, pipeline, and deployment version.

2. Instrument every pipeline stage for latency. Total request latency hides where time is actually spent. Stage-level latency will catch vector DB degradation, prompt bloat, and GPU contention before they become user-facing problems.

3. Use response quality proxies. Refusal rate, response length distribution, and repetition detection catch failures that no error code reports. Set baselines and alert on deviations.

4. Detect drift continuously. Embedding drift and output distribution shift are the slow-burn failures of AI systems. They degrade quality over days or weeks and are invisible to point-in-time checks. Monitor them continuously and trigger reviews at statistical thresholds.

5. Trace multi-step workflows. Aggregated metrics smooth out the pathological patterns that only appear at the trace level. Tool call loops, context window wastage, and repeated failures are invisible in aggregate. You need traces to see them.

6. Alert on error budget consumption, not every violation. Use statistical baselines adapted to each model and pipeline. Downgrade alerts during deployments. Reserve pages for real incidents, not normal variance.

7. Match your monitoring investment to your scale. Don’t build a custom observability platform at 10K requests per day. Do build cost tracking and trace-based monitoring before you hit 1M per day. The cost of missing an incident scales with your traffic, and the cost of monitoring scales with your complexity. Keep them in balance.

The systems we monitor today look nothing like the systems we monitored five years ago. The principles of good observability — instrumentation, aggregation, alerting, and debugging — are the same. But the metrics, the signals, and the failure modes are entirely new. The teams that adapt their monitoring to match the unique characteristics of AI systems will catch incidents that everyone else finds out about from the finance report or the support ticket.