Production Monitoring for LLM Applications

We shipped a customer-facing chatbot last quarter. In staging it was flawless — responses were coherent, latency hovered around 800 milliseconds, and the evaluation suite gave us green checks across the board. Within four hours of production traffic, three things happened simultaneously: response times spiked to 12 seconds under load, the model started answering “I don’t have that information” to questions it had answered perfectly during testing, and our token bill ran 40% over the projected daily budget. Every single issue was invisible in staging.

This isn’t a bug report. It’s a pattern. LLM applications exhibit failure modes that traditional software monitoring was never designed to catch, and they manifest at production scale in ways that staging environments cannot replicate. The non-deterministic nature of model outputs, the dependency on external API services, and the cost profile that scales with every single request all demand a monitoring approach that goes far beyond the classic “is it up?” dashboard.

Here is the practical monitoring setup we have landed on across multiple production LLM deployments — the metrics, traces, alerts, and dashboards that catch problems before users report them, and the tooling choices that make each piece achievable.

The Monitoring Pyramid for LLM Applications

In traditional backend monitoring, we think about a single stack — CPU, memory, error rate, latency. LLM applications add three new layers on top of that foundation, each with its own data sources, thresholds, and failure modes. We structure our monitoring as a pyramid with four tiers.

Each tier builds on the one below it. You cannot diagnose a quality issue without first knowing that the operational layer is healthy. And you cannot justify observability spend to a VP without connecting it to business metrics. Let us walk through each tier.

Tier 1: Operational Metrics (the Basics, Still Essential)

The foundation never changes. Every LLM-powered endpoint needs the same operational monitoring we have used for十五年 (okay, let’s say fifteen years) of web services:

• Latency p50, p95, p99 — measured from when the user’s request hits your server to when the response body is fully written. But with LLMs we slice this finer: time-to-first-token (TTFT) versus total response time. A 30-second total latency is acceptable for a long document summary if the user sees tokens streaming within the first second. A 30-second total latency with a 25-second TTFT is a disaster.
• Error rate by HTTP status — standard 5xx tracking, but also 429 (rate limited) and the increasingly common 529 (API overloaded). We track these per model provider and per endpoint.
• Throughput — requests per minute, with separate tracking for streaming versus non-streaming endpoints. Streaming endpoints can hide throughput issues because the connection stays open, masking queuing delays.
• Availability — uptime of the LLM API provider itself, measured via synthetic health checks every 60 seconds. When OpenAI, Anthropic, or your self-hosted model server has an incident, you want to know before your users’ “this is broken” emails arrive.

Concrete thresholds we use: p95 latency alarm at 5 seconds for chat endpoints, error rate alarm at 2% for any 5xx or 429 response, availability alarm at 99.5% over a 5-minute window. These are not aspirational — they are the numbers where users start abandoning your application.

Tier 2: LLM-Specific Metrics

This is where LLM monitoring diverges from traditional monitoring. Every request to a model produces a set of metrics that have no equivalent in conventional API monitoring.

Token usage per request. We log prompt tokens and completion tokens for every single model call. This data feeds into cost tracking, but more importantly it catches regression issues: a prompt that was 500 tokens in staging ballooning to 4,000 tokens in production because of a runaway context-building loop in the retrieval step.

Cost per request and cost in aggregate. We compute cost per request using the model’s per-token pricing and track it as a metric. This lets us set budget alerts — for example, a daily spend ceiling with a PagerDuty notification if we exceed 80% of the daily budget by 2 PM. The goal is to catch the “someone deployed a greedy prompt to GPT-5.5 instead of GPT-4.1-mini” scenario before it costs a month’s budget in an afternoon.

Model latency versus total latency. This is a critical distinction. Total latency is what the user experiences. Model latency is what the API provider charges for. The gap between them is your application code — retrieval, prompt assembly, response parsing, post-processing. We track both and alert when model latency is normal but total latency is high, because that points to your code, not the model provider.

Prompt and completion lengths. We log the raw lengths as metrics and track their distributions over time. A sudden shift in prompt length often indicates a retrieval pipeline change or a prompt template bug. A shift in completion length might indicate a system prompt change that the model is interpreting differently.

Rate limit hits. Every 429 response gets logged with the retry-after header value and the endpoint that was hit. We track rate limit hit rate per provider and per model. This data drives capacity planning and provider selection — if you are hitting rate limits daily on GPT-5.4, you either need a higher tier, a fallback model, or a caching layer.

Tier 3: Quality Metrics

Quality monitoring is the hardest tier and the one most teams skip. It is also the tier that catches the failures that users notice most.

Hallucination rate. We sample a percentage of production responses — typically 5-10% for high-traffic endpoints — and run automated evaluation against the source context. For RAG applications, this means checking that every factual claim in the response is grounded in one of the retrieved documents. We use LLM-as-judge evaluations with a structured rubric, not a single prompt asking “does this look right?” Our confidence threshold for flagging a hallucination is rigorous: we require two independent evaluator runs to agree before triggering an alert.

Task completion rate. For task-oriented applications (chatbots that book meetings, generate reports, answer support tickets), we measure whether the user’s goal was achieved. This requires instrumenting your application to emit a completion signal — the user clicked “confirm,” the ticket was resolved, the document was downloaded. We track this as a ratio of completed tasks to initiated tasks and alert on a 5% drop over a 24-hour window.

User feedback signals. We collect explicit feedback (thumbs up/down, star ratings) and implicit signals (conversation abandonment, response copy-paste, follow-up question cadence). Implicit signals are often more honest than explicit ones — users rarely click a thumbs-down button, but they frequently abandon a conversation that is not helping them. We track “conversation abandonment rate” as the percentage of conversations where the user sends no message within 10 minutes of the last bot response.

Output coherence proxies. Not every application needs a full evaluation pipeline. A reasonable proxy for response quality is the ratio of user messages to assistant messages, or the average length of assistant responses relative to a baseline. Sharp changes in these proxies rarely indicate improvement — they usually indicate a prompt change that broke something.

Tier 4: Business Metrics

The top of the pyramid connects monitoring to outcomes that the rest of the organization cares about.

• Conversion rate. If the LLM feature drives a conversion action (signup, purchase, subscription), we track conversion rate before and after the feature was introduced. A drop suggests the model is creating friction.
• Retention. We measure day-7 and day-30 retention for users who interact with the LLM feature versus those who do not. If the LLM feature reduces retention, it is actively harming the product.
• Revenue impact. For paid LLM features, we track gross margin per user: revenue from the feature minus the model cost to serve that user. This is the number that determines whether your feature is a business or a charity.
• Support ticket volume. An increase in support tickets that mention “chatbot,” “AI,” or the specific feature name is an early indicator of quality problems. We track this weekly and correlate it with deployment timestamps.

Setting Up Distributed Tracing for LLM Calls

Metrics tell you something is wrong. Traces tell you what, exactly, is wrong. For LLM applications, a single user request can fan out into five or more sub-operations: authentication, retrieval embedding, vector database query, prompt assembly, model API call, response processing, and post-generation guardrail check. When something fails, you need to see the entire chain.

We apply distributed tracing with the following spans for every user-facing LLM request:

User Request
  -> Auth Check (span)
  -> Query Understanding / Intent Classification (span)
  -> Retrieval:
       -> Embedding Generation (span)
       -> Vector DB Query (span)
       -> Re-ranking (span)
  -> Context Assembly (span - logs prompt length, number of chunks)
  -> Model Call:
       -> Request Queuing (span)
       -> Time to First Token (span)
       -> Streaming Duration (span - logs total completion tokens)
  -> Response Processing (span)
  -> Guardrail / Content Moderation (span)
  -> Response to User (span)

Each span captures:

• Duration and start time
• Token counts (for the model call span)
• The model ID, provider, and endpoint URL
• Error information with stack traces
• The truncated prompt (first 500 characters for privacy)

We propagate a trace ID from the user’s HTTP request through every downstream service using W3C Trace Context headers. This lets us correlate a user complaint (“the bot was slow at 3:15 PM”) with a specific trace, and from there to a specific model call, retrieval query, or bottleneck.

The most common trace-based finding we see: the model call itself is not the bottleneck. The retrieval pipeline — embedding generation plus vector database query — accounts for 60% or more of end-to-end latency in many RAG applications. Without traces, teams optimize model call latency while the real problem sits in the embedding step.

Alerting Strategy: What to Page On

Alert fatigue kills monitoring. If every anomaly pages someone, no one pays attention to pages. We separate alerts into two categories: page-worth and investigate-during-business-hours.

Page-Worthy (Immediate Response Required)

These get routed to PagerDuty with an escalation policy. For the quality cliff alert, we run an automated rollback of the last prompt or model deployment if the alert fires within 30 minutes of a deploy. This is aggressive, but a bad prompt can poison every user interaction in minutes.

Investigate During Business Hours

These alerts go to Slack or email. They signal investigation, not incident response. The key discipline is not letting the “investigate” bucket expand into the “page” bucket — every alert must have a clear action associated with it.

Building Dashboards for Different Audiences

A single dashboard that tries to serve everyone serves no one. Engineers, product managers, and executives need different views into the same data.

Engineering Dashboard: Traces and Per-Request Detail

• Latency heatmap by endpoint and model (refreshed every 60 seconds)
• Error rate breakdown by error type and provider
• Trace explorer with filtering by user ID, session ID, or trace ID
• Recent traces view showing the last 500 requests with latency, model, and status
• Token usage distribution (histogram of prompt and completion lengths)
• Model call breakdown — which models are handling what share of traffic

The engineering dashboard is where you go when a page fires. It needs to be fast, queryable, and detailed. We build it in Grafana with Loki for log aggregation and Tempo for trace storage. The critical feature is the ability to click from a latency spike on a graph directly to the traces that contributed to it.

PM Dashboard: Quality Trends and User Satisfaction

• Quality score over time (daily rolling average from automated eval)
• Task completion rate (daily, with 7-day and 30-day trend lines)
• User feedback aggregates (thumbs-up/down rate, abandonment rate)
• Feature usage metrics — daily active users, conversations per user, messages per conversation
• Top failing scenarios — the question categories or prompt patterns that consistently score low on quality eval
• Deployment timeline overlay — showing when prompt changes, model updates, or code deploys happened alongside quality metric changes

The PM dashboard answers one question: “Is the AI feature getting better or worse?” We build this in a BI tool (Metabase or Looker) with the same data source as the engineering dashboard, just a different aggregation level.

Executive Dashboard: Cost and Business Impact

• Total monthly LLM cost with breakdown by model and deployment
• Cost per conversation and cost per user
• Gross margin for paid AI features (revenue minus model cost)
• User retention comparison — LLM feature users vs. non-users
• Support ticket volume trend with “AI-related” tag highlight
• Uptime SLA attainment for the past 30 days
• Conversion rate impact — before and after the LLM feature launch

This dashboard gets shown in weekly all-hands. It stays high-level. If executives are asking questions that require them to drill into traces, your engineering dashboard is not surfacing the right abstractions.

The Tools Landscape

The monitoring and observability ecosystem for LLM applications is evolving rapidly. Here is how we evaluate the options.

LangSmith is the most comprehensive option if you are already using LangChain or building complex chains. Its tracing is deeply integrated with the LangChain execution model, and it provides built-in evaluation datasets and annotation queues. The trade-off: it pushes you toward LangChain’s abstractions, which can be constraining if you want direct control over model calls. It also adds significant latency if you sync traces in real-time rather than batch-exporting them.

Braintrust focuses on evaluation-first workflows. Its strength is the ability to run automated evaluations on production traffic, compare results across model versions, and track prompt iterations over time. The monitoring and alerting features have matured considerably alongside the evaluation tooling, and the combined capabilities are best-in-class. It is a strong choice if your primary concern is quality regression detection.

Arize started as a traditional ML observability platform and has expanded into LLM monitoring. Its strong suit is drift detection and embedding visualization — it can surface when the distribution of user queries has shifted, which is a leading indicator of quality problems. The setup is heavier than the alternatives (you typically run a collector agent alongside your application), but the drift detection capabilities are unmatched.

MLflow might already be in your stack if you are doing traditional ML experiment tracking. Its tracing and metrics capabilities provide a unified view of LLM calls alongside any traditional ML models you might be running. If you are already using MLflow for experiment tracking, its monitoring features are worth exploring before adding a separate observability tool. The tracing API is straightforward to instrument manually, and the cost tracking integrates with the same metrics infrastructure you may already use for model performance monitoring.

Our current stack is LangSmith for development and early production, with traces exported to Grafana/Tempo for long-term storage and cross-service correlation. We use MLflow for tracking traditional ML model performance alongside LLM metrics, since our platform runs both. For teams starting fresh with a single LLM feature, we recommend picking one tool based on your primary concern — evaluation (Braintrust), drift (Arize), or ecosystem compatibility (LangSmith or MLflow) — rather than trying to bolt all of them together.

Practical Takeaways

1. Start with the operational tier. Before you evaluate a single model output, make sure your p99 latency, error rate, and availability dashboards exist and have alerts. Everything else depends on these being reliable.

2. Log every model call. Token counts, model ID, provider, latency, and status code for every single request. This data is cheap to collect and invaluable for debugging. Store it in structured logs with trace IDs so you can correlate it with the rest of your application.

3. Sample for quality evaluation. You do not need to evaluate every response. A 5% sample with automated LLM-as-judge evaluation catches the vast majority of quality regressions. Increase sampling during deployments and decrease it during stable periods.

4. Build the dashboard chain. Three dashboards — engineering, PM, executive — each tuned to what the audience needs to act on. If your VP cannot answer “what is the business impact of the LLM feature?” from a dashboard, the chain is incomplete.

5. Test your monitoring before you need it. Simulate a model provider outage. Trigger a cost spike with a test deployment. Confirm that your pager actually fires. The worst time to discover that your alerting pipeline is broken is during the incident you built it to catch.

6. Review your prompt lengths regularly. This is the single highest-leverage monitoring practice that almost no one does. Prompt drift — where a prompt silently grows over time as features are added — is the most common cause of cost and latency regressions we see in production LLM applications. Put a prompt length alert on day one.

Production monitoring for LLM applications is not fundamentally different from production monitoring for any other distributed system. The same principles apply: measure what matters, alert on what requires action, and build dashboards for the decisions each audience needs to make. The difference is that the failure modes are subtler, the cost dynamics are faster, and the quality signals require more interpretation. Get the pyramid right, instrument your traces, and you will catch problems before your users do — which is, after all, the entire point.