LLM Metrics That Actually Matter

Here is a truth nobody wants to admit at a conference: most of the numbers teams track for their LLM products are actively misleading. We have watched teams ship features that made BLEU scores soar and user satisfaction tank. We have seen dashboards full of perplexity charts covering up the fact that the chatbot tells customers their order number is “probably 42” three times a week. If you are measuring the wrong thing, you are not just wasting time — you are making confident decisions based on confident lies. Let us fix that.

The Old Guard: Metrics That Need to Retire

Before we talk about what to measure, we need to talk about what to stop measuring. Traditional NLP metrics were designed for machine translation and text summarization research in the 2000s. They served a purpose then. They are actively dangerous now.

Perplexity measures how “surprised” a model is by a sequence of tokens. Lower is supposedly better. In practice, we have seen models with near-identical perplexity produce wildly different user experiences. Perplexity does not capture factuality, coherence, or whether the output answers the user’s question. One team we know optimized for perplexity and ended up with a model that was extremely confident, extremely fluent, and extremely wrong — the model equivalent of a smooth-talking salesman with no product knowledge.

BLEU counts n-gram overlap between generated and reference text. It was built for machine translation, where there is a “correct” answer. In a conversational AI product, a user asks “What is the return policy?” and there are a dozen perfectly good phrasings. BLEU punishes any phrasing that does not match the reference token-for-token. A model that says “You have 30 days to return items” scores lower than one that parrots the clunky reference phrase “Returns must be initiated within a period of 30 calendar days from the date of delivery.”

ROUGE shares the same flaw: it measures surface-form overlap, not semantic fidelity. A summary that captures all key points but uses different vocabulary scores worse than one that copies verbatim with minor deletions. We have watched teams reject a good summarization model because its ROUGE score was 2 points below a baseline that simply extracted the first three sentences of every document.

The common thread here is that these metrics measure form, not substance. They cannot tell you whether the output is true, useful, or safe. For a product that users interact with in natural language, those are the only things that matter.

These metrics still have a place — in research papers comparing base models, in ablation studies during fine-tuning. But they should never appear on a production dashboard. If your team lead asks why BLEU dropped by 3 points, you have a metrics problem, not a model problem.

The Metrics That Actually Predict User Behavior

After working with dozens of teams shipping LLM features to real users, we have landed on a core set of metrics that consistently correlate with user satisfaction and business outcomes.

Task-Specific Accuracy on Your Eval Set

This is the closest thing to a ground-truth metric, but it requires doing the work upfront. You need an evaluation set of realistic inputs paired with known-good outputs, and you need to grade model outputs against that set on a regular cadence.

The key word is “task-specific.” A general-purpose benchmark like MMLU tells you almost nothing about whether your customer-support chatbot correctly handles refund requests. Build your own eval set. Start with 200 examples covering the common cases and 50 covering edge cases. Grade them manually or with a judge model you have validated against human ratings. Run this eval every time you change your prompt, your model, or your retrieval pipeline.

Aim for 95%+ accuracy on common cases and 85%+ on edge cases before shipping. These thresholds come from observing user behavior: below these numbers, users start abandoning the interaction or escalating to a human within three turns.

Hallucination Rate

This is the single most important metric for any LLM feature that surfaces factual information, and it is also the hardest to automate. Hallucination is not binary — it exists on a spectrum from “subtle implication that is technically unsupported” to “completely fabricated citation with a real-sounding author name.”

The standard we recommend: flag any output that contains a factual claim not supported by your provided context or known ground truth. The acceptable threshold depends on your domain:

• Customer support (retrieval-augmented): Under 0.5% of outputs should contain unsupported claims. Users who encounter a hallucination are 3x more likely to churn within the same session.
• Code generation: Under 1% for common patterns, under 5% for complex or framework-specific tasks. A hallucinated API call that looks real but does not exist costs developers 15-45 minutes of debugging time.
• Content summarization: Under 1% for factual inaccuracies. A hallucinated detail in a summary erodes trust in a way that takes 3-5 accurate summaries to rebuild.
• Creative or brainstorming features: Higher rates are acceptable, but flag them. Users of creative tools are more forgiving of factual errors, but they still lose trust when they notice them.

Measure hallucination rate by sampling 10-20% of outputs and running them through a judge model with a structured rubric. We have found that a GPT-5-level judge, when given clear context and a rubric, agrees with human raters on hallucination detection about 92-95% of the time.

Latency at p95 and p99

Speed is a feature. Every 500ms of additional latency above a 2-second threshold reduces user engagement by roughly 5-8%. But average latency is a trap — it hides the tail.

The p95 and p99 latencies tell you what actual users experience during peak loads. If your p50 is 800ms but your p99 is 12 seconds, then 1 in 100 users is having a terrible experience. In a product serving 100,000 requests per day, that is 1,000 frustrated users daily.

For streaming applications, track both time-to-first-token and inter-token latency variance. A stream that starts fast but stutters is often worse than a slightly slower steady stream. We have seen user satisfaction scores drop 15% when inter-token latency varies by more than 100ms, even when average throughput is fine.

Cost Per Successful Interaction

This is the metric that makes executives happy and keeps your budget alive. Calculate it as total inference cost divided by the number of interactions the user rated as successful.

A few real data points from production systems we have observed:

• Simple Q&A (RAG): $0.003 - $0.008 per successful interaction with a frontier model, $0.0005 - $0.002 with a smaller fine-tuned model.
• Code generation: $0.01 - $0.05 per successful completion, heavily dependent on output length and retry rate.
• Multi-turn chat (5+ turns): $0.02 - $0.10 per session, dominated by the cost of generating the longer context.

If your cost per successful interaction exceeds $0.10 for a simple task, you have an architecture problem — likely over-retrieval, overly verbose prompts, or using a model that is too large for the task.

Track this metric over time and watch for degradation. When a prompt change adds 200 tokens of system instructions and the cost per success goes up 30% without a corresponding accuracy improvement, you have a signal to revert.

User-Reported Quality Scores

The metrics above are all proxies. User-reported quality scores are the closest thing to ground truth. The most reliable signals we have seen are:

Thumbs up / down ratio. Simple, binary, and surprisingly robust. Track it per session, per model version, and per input category. A 1:10 thumbs-down-to-thumbs-up ratio is baseline acceptable. Below 1:5, something is wrong. Above 1:20, you are probably undersampling — happy users rarely tap thumbs up, but frustrated users reliably tap thumbs down.

Copy rate. When a user copies the model’s output, it is a strong positive signal — they found the answer useful enough to take action on. When they type a follow-up that starts with “No, I meant…” or “That is wrong because…”, it is a strong negative signal. Track these implicitly with front-end events; they require no extra user effort and are highly reliable.

Share rate. Users who share a model’s response are the highest-intent users. If share rate drops after a model update, investigate immediately.

Correction rate. The percentage of interactions where the user explicitly corrects the model or asks it to try again. This is your canary in the coal mine. A correction rate above 15% indicates a fundamental quality problem that no amount of prompt engineering will fix.

Task Completion Rate

This is the metric that translates model quality into business outcomes. Did the user get what they came for?

Define task completion differently for each feature:

• Support chatbot: Did the user resolve their issue without escalating to a human? Measure as percent of sessions that end without a “talk to agent” click.
• Code assistant: Did the user accept the generated code (copy, paste, run)? Measure as percent of generations followed by an acceptance event within 5 minutes.
• Content generator: Did the user publish or export the generated content? Measure as percent of generation sessions that end with an export action.

Baseline acceptable completion rates: 70-80% for support chatbots, 60-75% for code generation, 50-65% for content generation. These numbers seem low, but they reflect the reality that users experiment with LLMs — not every interaction is a serious attempt.

Track completion rate by user segment. Power users (50+ interactions per week) should see completion rates 15-20 points higher than new users. If they do not, your model is not adapting to recurring use patterns.

Refusal and Block Rate

A model that never refuses is unsafe. A model that refuses too often is useless. The sweet spot depends on your domain and risk tolerance.

Measure two things:

Refusal rate. The percentage of inputs where the model declines to respond. For most applications, 0.5-2% is the right range. Below 0.5%, you are probably not catching enough unsafe inputs. Above 5%, you are frustrating users with legitimate requests that trigger overly broad safety filters.

Block rate. The percentage of inputs rejected before they reach the model, by your input guardrails. This is a separate metric because it measures your safety infrastructure, not the model itself. A block rate above 10% suggests your guardrails are too aggressive. Below 0.1% suggests you have no real guardrails at all.

Both metrics should be tracked per input category. We have seen products where 15% of finance-related questions were blocked — a product-killing number for a fintech app — while the overall block rate looked fine at 2%. Category-level tracking catches these problems.

Consistency Score

Users expect the same input to produce roughly the same output. When a model gives two different answers to the same question, trust erodes. We measure consistency by taking the same input, running it through the model 5-10 times at your production temperature, and measuring semantic similarity of the outputs.

Aim for 90%+ consistency on factual questions and 70%+ on creative tasks. Below 50% on any task type, users start noticing. The most dangerous failures are subtle: two answers that say the same thing but disagree on a specific number or date. Track semantic consistency with an LLM judge that checks for factual contradictions, not lexical overlap.

Metric Cascades: Your Leading Indicators

No single metric tells the full story. The magic is in understanding how metrics relate to each other — what we call metric cascades.

Here is the cascade we see most consistently:

Latency p95 increases -> User patience drops -> Users rephrase inputs -> Average input length increases -> Cost per interaction rises -> Task completion rate drops -> Thumbs down ratio increases.

The signal appears in latency data 3-7 days before it shows up in satisfaction scores. If you are only watching satisfaction scores, you are already a week late.

Build your dashboards around cascades, not individual metrics. When latency p95 crosses your threshold, automatically review cost per interaction and task completion rate. When hallucination rate ticks up, audit the retrieval pipeline and prompt system instructions.

Building a Dashboard People Actually Use

The best metric in the world is useless if nobody looks at it. Here is what we have learned about dashboards teams actually check versus those that become Datadog graveyards.

Limit the dashboard to 7-9 metrics. Every metric beyond that reduces the chance anyone looks at it by roughly 10%. If you cannot explain a metric’s presence in one sentence, remove it.

Put the three most important metrics at the top, left-to-right. For most LLM features, those are task-specific accuracy, hallucination rate, and cost per successful interaction. Visible without scrolling. Everything else goes below.

Use alerts, not color coding. Red-yellow-green dashboards train people to ignore everything that is green. Set up alerts for metric thresholds. An alert that fires at the same time every day gets muted within a week.

Track the cascade, not the count. Do not show a latency chart alone. Show it alongside task completion rate and cost per interaction, on the same time axis, with the correlation annotated. When a viewer sees latency spike and completion rate drop on the same day, they start to understand the system.

Make it real-time enough, but not real-time. A 1-hour lag is fine for most teams. Real-time dashboards create noise — every latency spike during a deployment looks like a crisis. Give data time to stabilize before surfacing it.

The Trap: When Optimizing a Metric Makes the Product Worse

Every metric can be gamed. When a metric becomes a target, it ceases to be a good measure. We have seen this pattern repeat:

Optimizing for latency. A team pushes model responses to be faster and faster, and latency drops by 40%. Task completion rate drops by 15%. The model is faster because it is outputting shorter, less complete answers. Users now have to ask follow-ups they did not need before. The “faster” system is actually slower for the user.

Optimizing for cost per interaction. A team switches to a smaller model and cost drops by 60%. Consistency score drops by 25%. Users start getting wrong answers. The cheaper system costs more in lost trust and support escalations.

Optimizing for thumbs up ratio. A team tunes the model to be more agreeable. Refusal rate drops to near zero. Hallucination rate goes up because the model now says yes to everything, including requests where it should push back. Thumbs up ratio goes up for a week and then collapses as users realize the answers are unreliable.

The antidote is not to stop measuring — it is to measure the right things in the right combination. Never optimize a single metric. Always optimize against a weighted combination, and re-weight it based on user outcomes at least once a quarter.

Practical Thresholds: What Good Looks Like

Here are the production-tested thresholds we share with every team we work with. These are not theoretical — they are drawn from observing millions of LLM interactions across customer support, code generation, content creation, and internal tooling.

These thresholds shift by domain. A medical diagnosis assistant needs hallucination rates below 0.01% and may need a higher refusal rate by design. A creative writing tool can tolerate higher hallucination rates but needs p95 latency under 1 second. Start here and tune as you collect data.

What We Actually Do Now

Here is the practical workflow we recommend to every team shipping an LLM feature.

Week 1: Establish a task-specific accuracy baseline. Build your eval set. Run 200 examples through your model. Grade them. If it is not at 85% on common cases, do not ship yet.

Week 2: Set up production monitoring for latency and cost. Get p95, p99, and cost-per-interaction into whatever dashboard your team already uses. Structured logging and a standard dashboard will get you 80% of the way there.

Week 3: Add hallucination monitoring. Sample 10% of outputs. Run them through a judge model. Set an alert that fires when the 7-day rolling average crosses 0.5%.

Week 4: Ship and watch user-reported quality. Add front-end events for thumbs up/down, copy, and correction. Correlate these with your automated metrics. If the correlations are weak, your automated metrics are wrong.

Month 2: Build your metric cascade. Connect the metrics so a change in one triggers a review of others. Automate the alerts. Switch to weekly reviews.

Month 3: Audit your metrics. Check whether optimizing your metrics actually improved user outcomes. If not, change them. This step is the one almost everybody skips, and it is the most important one.

The teams that succeed with LLMs are not the ones with the smartest prompts or the largest models. They are the ones that measure what matters, ignore what does not, and have the discipline to change their metrics when the data tells them they are wrong. Start with the metrics in this article, cut anything that does not predict a real user outcome, and revisit your dashboard every quarter like you are looking at it for the first time.