How Leading AI Companies Test Models

If we asked you when the last time a bad model made it to production and how many users saw it before anyone noticed, what would your answer be? At Anthropic, OpenAI, and Google DeepMind, that number is zero more often than not, and when it isn’t zero it is single digits — because they have built evaluation infrastructure that catches failures long before a single user types a prompt. What separates teams that ship reliable models from teams that ship hope is not talent. It is process, infrastructure, and a willingness to measure things that are uncomfortable to look at. Let us walk through exactly how the best AI companies test their models, and more importantly, which of their practices any team can adopt — whether you are ten people, five, or just two with a shared API key.

The Two Layers of Evaluation Nobody Talks About

There is a fundamental divide in model evaluation that most writing in this space ignores. The leading labs operate at two distinct layers, and confusing them is where most teams waste effort.

Layer one is model-level evaluation. This happens at the research lab. A new checkpoint, a new architecture, a new training recipe. We want to know: is this model better than the last one? Does it have catastrophic safety failures? Did it forget how to do basic arithmetic? This evaluation happens on massive GPU clusters with standardized benchmarks. It measures raw capability.

Layer two is application-level evaluation. This happens when we wrap a model in a product — a chatbot, a code assistant, a document summarizer. We want to know: does this model actually help users accomplish their task? Does it follow our prompt instructions reliably? Does it refuse too often or not often enough? This evaluation happens on our application data with our success criteria.

The labs are exceptional at layer one. Most teams building on top of their APIs over-index on layer one metrics (benchmark scores, leaderboard positions) and under-invest in layer two. Here is the uncomfortable truth: benchmark scores tell us almost nothing about whether a model will work well as a customer support triage agent. The labs know this. Their internal evaluations mix both layers, with more weight on application-level signals as a model gets closer to shipping.

What the Big Labs Actually Measure

Let us get specific about what each major player tracks internally. This comes from published research, blog posts, conference talks, and conversations with engineers at these companies.

Anthropic

Anthropic’s evaluation philosophy is shaped by their constitutional AI approach and their relentless focus on the helpfulness-harmlessness tradeoff. Their evaluation stack includes:

Anthropic publishes their evaluation methodology in each model release’s system card. What stands out is the volume: for Claude 3.5 Sonnet, they ran over 10,000 automatic evaluation scenarios per training checkpoint. Not at release — continuously throughout training to catch regressions the moment they emerge.

They also maintain “evals as tests” — a philosophy borrowed from software engineering where evaluation scenarios are treated as unit tests that must pass before a model is deployable. If a new training run drops performance on a critical safety evaluation by even 2%, that run is discarded or the recipe adjusted. This is the closest analog to CI/CD in the AI world today.

OpenAI

OpenAI’s evaluation approach is more product-driven, which makes sense given their broad API surface and consumer products like ChatGPT. Their known practices include:

Pre-training evals. During training, they run lightweight evaluations every few thousand steps. If a model forgets how to format JSON after a training update, they know within hours, not weeks.

Post-training evals. After RLHF or fine-tuning, comprehensive evaluations run across hundreds of dimensions — coding (HumanEval, SWE-bench), factuality (SimpleQA, internal datasets), safety (refusal rates, jailbreak resistance), and instruction following. These are automated and produce org-wide dashboards.

Production shadow evaluation. This is the golden practice too few teams copy. Before deploying a new model, OpenAI runs it in “shadow mode” — the new model generates responses alongside the current production model, but users never see them. Instead, they compare the outputs: which conflict? Which are preferred by automated judges? Deployment is blocked if regressions appear.

This is the practice to steal first. It costs engineering time, but we will show you a minimal version any team can set up in an afternoon.

Google DeepMind

DeepMind brings the research rigor. Their evaluation practices reflect their origins in academic publishing and their current mandate to ship real products at Google scale.

Standardized benchmarks with strict reproducibility. DeepMind publishes exact evaluation protocols — down to the random seed, prompt template, and decoding parameters. When DeepMind releases a Gemini scoresheet, you can reproduce every number if you have the same hardware.

Multi-task evaluation across modalities. Given their multi-modal focus, DeepMind evaluates separately for text, image, audio, and code. Each modality has its own pipeline and regression thresholds.

Reward model evaluation. DeepMind evaluates their reward models against human preferences and tracks agreement rates over time. This is underappreciated: if your reward model diverges from human judgment, your trained model optimizes for the wrong thing. Monitoring agreement is like checking that your compass still points north.

Red Teaming: The Practice That Separates Professionals from Amateurs

Every major lab runs systematic red teaming, but the approaches vary significantly. Here is what the best ones do that others do not.

Continuous red teaming, not one-off events. The naive approach is hiring a red team for two weeks before launch. The professional approach is maintaining an ongoing red team that tests each training run, prompt update, and system change. OpenAI and Anthropic both maintain internal red teams and supplement with external contractors on specific domains.

Domain-expert red teaming. Generic red teaming catches generic failures. Domain-expert red teaming catches failures that would actually harm users. Anthropic contracts with doctors for medical scenarios and lawyers for legal scenarios — finding failures that general-purpose testers would miss.

Automated red teaming. All three labs use automated red teaming where one LLM generates test cases designed to fail another. Results are triaged by humans. This scales to tens of thousands of test cases per day, and open-source tools like garak get any team 80% of this capability for free.

Structured red teaming taxonomies. The labs do not just ask red teamers to “find bugs.” They provide structured taxonomies of failure modes to test against:

• Bias and fairness: Demographic associations, stereotyping, disparate performance across languages
• Safety: Toxic content, dangerous capabilities, instructions for harm
• Reliability: Contradictory responses, hallucination patterns, number-of-X counting failures
• Security: Prompt injection, jailbreaking, training data extraction
• Alignment: Refusal rates (too high or too low), sycophancy, capability sandbagging

If your red team has no taxonomy, they write. If they have a taxonomy, they hunt.

The Pre-Deployment Checklist

Based on what we have learned from the practices of leading labs, here is a consolidated pre-deployment checklist. How many of these can we honestly check off?

[ ] All known failure modes from the previous model version are regressed
[ ] Automated evals pass on 95%+ of scenarios across all risk categories
[ ] Production shadow evaluation shows no regressions versus current model
[ ] Red team found no critical-severity issues in the last 72 hours
[ ] Latency p95 and p99 are within SLA for the target use case
[ ] Cost per inference is within budget
[ ] Refusal rate on legitimate prompts is below threshold (<5%)
[ ] Jailbreak success rate is below threshold (<1%)
[ ] Response quality (human eval) is not worse than current production model
[ ] Monitoring metrics and dashboards are configured for the new model
[ ] Rollback plan exists and has been tested
[ ] Communication plan exists for any known capability changes

OpenAI and Anthropic do not ship a model that fails this checklist. Neither should any team shipping AI products.

What Scales Down to Small Teams

Here is the practical part. We do not need a team of fifty to implement these practices. We need the right 20% of the practices applied consistently.

Build an Eval Dataset, Not an Eval Vibe

The single highest-leverage thing any team can do is build a structured evaluation dataset. Not “we played with the model for a few hours and it seemed fine.” A structured dataset with:

• 200-500 input-output pairs covering our key use cases
• Explicit criteria for what constitutes a pass or fail for each pair
• Multiple variants of the same prompt to catch prompt-sensitivity
• Edge cases identified through user behavior analysis

We do not need to build this all at once. Start with 50 examples. Add 10 per week. In three months, we have 200-plus examples. In six months, we have a dataset that will catch most regressions before users ever see them.

Automate Our Eval Runs

Manually running evaluations is a recipe for skipping them. Automate with our CI/CD pipeline. Every commit that changes a prompt, a model version, or a system prompt triggers an automated eval run. Push to a branch? Eval runs. Open a PR? Eval blocks merge if scores drop below threshold.

Tools that make this practical include:

• Pytest with promptfoo for simple eval-as-test setups
• LangSmith or Weights and Biases for tracking eval results over time
• GitHub Actions or GitLab CI for running evals on every commit

If our eval takes more than 5 minutes or produces output we have to manually interpret, we will skip it. Aim for evals that run in under 2 minutes and produce a single pass-fail score per category.

Production Shadow Mode

Shadow mode is not exclusive to the big labs. Here is the simplest version: deploy our new model to a separate endpoint, route 1% of production traffic to it alongside our current model, log both responses, and compare them in a dashboard the next morning. If the new model shows more hallucinations, higher refusal rates, or worse user satisfaction scores, revert it before rolling out to 100%.

The cost is trivial — a second API endpoint and a comparison script. Most teams catch something their offline evaluation missed on the first try.

Systematic Red Teaming for Small Teams

We probably cannot afford a dedicated red team. But here is what we can do.

First, build a red teaming prompt library. Collect 50 to 100 adversarial prompts from known attack patterns — jailbreaks, prompt injections, role-playing attacks. Keep them in a version-controlled file. Run them against every model update.

Second, rotate red teaming duty. Have one team member per sprint spend two hours trying to break the model. The rotation distributes the cognitive load and brings fresh perspectives.

Third, use automated red teaming tools. Garak is an open-source tool that generates thousands of adversarial inputs. It is not as good as a human domain expert, but it is dramatically better than nothing.

The marginal benefit of the first hour of red teaming per week is enormous. The marginal benefit of the tenth hour declines. Find our team’s optimal investment level and automate the rest.

The Metrics the Best Teams Track That Average Teams Ignore

Everyone tracks latency, token count, and basic accuracy. Here is what the labs track that most teams do not.

Refusal rate on legitimate prompts. If our model refuses to help with benign requests, it is not safe — it is broken. Track this metric separately from refusal rate on harmful prompts. The ratio between the two tells us how well-tuned our safety system is. A model that refuses nothing dangerous and refuses nothing legitimate is the goal, and tracking this ratio is the only way to know whether we are approaching it.

Syophancy score. How often does the model agree with the user even when the user is wrong? Labs measure this by prompting with deliberately incorrect premises and checking whether the model pushes back. High sycophancy correlates with lower factual reliability. Some models have shown rates above 40% — agreeing with a false premise nearly half the time.

Across-group consistency. Does the model perform equally well for users from different demographics and dialects? The labs measure this through stratified evaluation across demographic slices. We can approximate it by ensuring our evaluation dataset reflects our actual user base, not just our engineering team.

Instruction drift over context length. How does performance degrade as conversations get longer? Most teams evaluate on single-turn prompts. In production, conversations routinely go 50 turns deep. The best teams measure performance at turn one, turn five, turn twenty, and turn fifty separately.

Edge case coverage. The labs track which scenarios catch regressions and which have never failed. A scenario that has never triggered a regression in six months is probably not challenging enough. They retire and replace scenarios periodically to maintain coverage quality.

The Evaluation Infrastructure Stack

Here is a simplified view of what the evaluation infrastructure looks like at a leading lab. Your version will be smaller, but the architecture should follow the same pattern:

Training Pipeline -> Checkpoint
    |
    v
Lightweight Eval (every 5K steps) -> Score History
    |
    v
Full Eval (each new candidate) -> Regression Report -> Fail? -> Discard checkpoint
    |
    v
Red Team Review (automated + human) -> Severity Report -> Critical fail? -> Discard checkpoint
    |
    v
Shadow Deployment -> Compare with production -> Regression? -> Block deployment
    |
    v
Canary Rollout (5% -> 20% -> 50% -> 100%) -> Monitor -> Regression? -> Rollback
    |
    v
Continuous Monitoring -> Dashboard -> Alerts -> Regression? -> Rollback or alert

The critical insight is the gating at every step. A checkpoint that fails lightweight eval never reaches red team review. A model that shows regressions in shadow mode never reaches production. Each gate adds confidence, and each operates independently.

What to Steal Right Now

Let us prioritize the five practices that give the highest return for the least effort.

Build our eval dataset today. Start with 50 examples. Add 10 per week. We can do this before our next team meeting.

Automate our eval runs. Attach them to CI. If we use GitHub, this is a YAML file and about 30 minutes of work.

Implement production shadow evaluation. Deploy new models to a second endpoint. Route 1% of traffic. Compare results each morning. This catches more regressions than any offline eval.

Track the right metrics. Measure refusal rates, sycophancy, and performance across actual user demographics, not just average accuracy.

Create a pre-deployment checklist. Make it concrete. Make it required. Block deployments that cannot pass.

When the Labs Get It Wrong

It would be dishonest to imply the labs have perfect evaluation. They do not. Here are the gaps that persist even at the highest level.

Evaluation distribution mismatch. Lab evaluations still overweight English-language, US-centric scenarios. Models perform differently for non-English users and cultural contexts outside the training distribution. This is a known gap every lab is working on.

Automated judge bias. Using LLMs to evaluate LLMs introduces systematic biases — they prefer longer responses, more confident-sounding responses, and responses matching their training data. Labs adjust for this with calibration techniques but cannot eliminate it entirely.

Ghost in the eval data. When a model has seen evaluation data during training through web-scale crawling, the scores become meaningless. Detecting and eliminating data contamination is an ongoing arms race.

What evals miss. No evaluation suite catches emergent capabilities. If a model develops a new capability during training, it will not be caught until someone designs a test for it. This is the fundamental limitation: we can only test for what we think to test for. Labs mitigate this with open-ended red teaming, but the gap remains.

Practical Takeaways

Here is what we want you to take away from this exploration of how the leading labs test their models.

Steal their infrastructure, not their scale. The evaluation practices that make the biggest difference — structured datasets, automated regression testing, shadow evaluation, systematic red teaming — scale down to any team size. What does not scale is running 10,000 evaluations per training step. We do not need that.

Invest in evaluation early. Every team we have worked with regrets not building their evaluation infrastructure sooner. No team has ever regretted building it. The cost of a bad model in production — lost users, lost revenue, lost trust — is orders of magnitude higher than the cost of catching it in evaluation.

Treat evaluation as a product, not a project. It is not something we build once and forget. Our evaluation infrastructure needs continuous investment: new scenarios, updated metrics, better automated judges, broader coverage. The labs treat evaluation as a first-class engineering discipline with dedicated teams and continuous roadmap. We should too.

Start before we feel ready. Our first evaluation dataset will be small and imperfect. Our first automated eval pipeline will have gaps. Our first shadow deployment will miss things. That is fine. Every one of these is better than shipping a model to production with no evaluation at all. The best time to start building evaluation infrastructure was six months ago. The second best time is right now.