Regression Testing for AI Products

We shipped a model update last quarter that looked great in offline eval — higher accuracy, lower perplexity, faster inference. Two weeks later, our support ticket volume had doubled. Users were getting confidently wrong answers to a specific class of questions the old model happened to handle well. We had no safety net, no automated gate, and no way to detect the regression until our customers told us about it. That hurt.

The problem is fundamental to AI products: traditional software either works or it doesn’t. A button click either opens a modal or throws an exception. An API endpoint either returns 200 or 500. But AI systems degrade subtly. The model still returns answers — they’re just worse answers. Slightly less relevant. Slightly more verbose. Occasionally hallucinated. The system appears to be running fine right up until the point it isn’t.

This article is about the practical infrastructure we’ve built to catch regressions in AI products before they reach users. We’ll cover model upgrades, prompt drift, pipeline degradation, golden dataset maintenance, and the CI/CD patterns that make regression testing a routine part of shipping — not a post-mortem activity.

Why AI Products Regress Differently

Traditional software regressions are usually binary: a feature breaks because someone changed a code path that an existing test didn’t cover. The fix is straightforward — revert the change or patch the missed edge case.

AI regressions are more insidious. Here is what we’ve seen cause them in practice:

The common thread is that nothing crashes. Every component returns valid output. The system as a whole just performs worse, and without explicit evaluation you will not notice until user metrics decline.

We learned this the hard way when a minor prompt tweak (“be concise”) started producing one-sentence answers to complex technical questions after a background model update changed how “concise” was interpreted. The prompt change was rolled out three weeks before the model update, and the interaction between them was invisible to any single test.

What Regression Testing Looks Like for AI

Regression testing for AI products is not the same as unit testing. You cannot assert that the output equals a specific string. Instead, you assert that the output satisfies quality constraints — and you measure those constraints across a representative test set.

We structure our regression testing around three layers:

Layer 1: Eval-gating model upgrades. Before any model version switch (new fine-tune, updated API model, different provider), we run a standardized eval suite against both the current and candidate model. The upgrade only proceeds if the candidate meets or exceeds the current model on every critical metric.

Layer 2: Prompt regression suites. Every prompt template change triggers a regression run against a fixed test set of 200-500 examples. This catches interactions where a prompt change meant for one use case degrades another.

Layer 3: Production shadow evaluation. The candidate model runs alongside the production model on a sample of live traffic. Outputs are compared offline before any user-facing switch happens.

Eval-Gating Model Upgrades

An eval gate is a script that does roughly this:

def should_upgrade(current_model, candidate_model, eval_dataset, metrics):
    current_scores = evaluate(current_model, eval_dataset, metrics)
    candidate_scores = evaluate(candidate_model, eval_dataset, metrics)
    
    for metric in metrics:
        delta = candidate_scores[metric] - current_scores[metric]
        threshold = metric_configs[metric]["regression_threshold"]
        if delta < -threshold:
            return False, f"{metric} regressed by {delta:.3f}"
    
    return True, "All metrics pass"

The key design choice is the threshold. Zero-tolerance — “never regress on any metric” — sounds safe but practically blocks every upgrade because noise in evaluation always produces small deltas. We use thresholds calibrated to the observed variance of each metric across three evaluation runs of the current model.

These thresholds live in our repo alongside the evaluation config, version-controlled and reviewed like any other config change.

Prompt Regression Suites

Prompt changes are the most common source of regressions in our experience — more common than model upgrades. A prompt change is cheap and easy, so teams make them frequently. Each one carries risk.

Our rule: any commit that touches a prompt template file must pass the prompt regression suite in CI. The suite runs the prompt(s) against a fixed test set and compares outputs to stored references using semantic similarity thresholds. If the new prompt changes behavior on more than 10% of examples beyond a similarity threshold, the build fails and a human reviews the diff.

We use pytest with a custom mark for this:

@pytest.mark.prompt_regression
class TestCategorizationPrompt:
    def test_output_format_preserved(self, prompt_regression_fixture):
        result = run_prompt("categorization", prompt_regression_fixture.input)
        assert is_valid_json(result.output), "Output format changed"
        assert semantic_similarity(
            result.output, 
            prompt_regression_fixture.expected
        ) > 0.85, "Output semantics drifted"

The test set for prompt regression suites is curated manually — every example is hand-labeled with the expected output characteristics. This is expensive to build but relatively cheap to maintain. We add 5-10 new examples each sprint from production edge cases.

Golden Dataset Maintenance

Your regression test set is only as good as its relevance to production traffic. A golden dataset that is six months old will test things your model no longer does and miss things your users actually need.

We maintain our golden datasets with a quarterly refresh cycle:

1. Sample production inputs from the last 30 days, stratified by user segment and input type.
2. Deduplicate against existing test examples using embedding similarity (threshold: cosine similarity > 0.95).
3. Label new examples with expected output characteristics — not the exact output, but the properties it should satisfy (format, tone, factual constraints, refusal boundaries).
4. Remove stale examples where the task or product behavior has changed enough that the expected output is no longer well-defined.
5. Re-run baseline evals on the refreshed set to establish new performance baselines.

MLflow’s experiment tracking is useful for comparing model versions side-by-side on your eval metrics, making it easier to detect when a model upgrade causes regressions on specific subsets of your data. We log every eval run — including our golden dataset evaluations — as experiments, with the dataset version as a tag. When a new model fails on a specific subset, we can trace back through the experiment history to see whether that subset has always been problematic or whether the regression is new.

CI/CD Integration Patterns

Regression tests are only useful if they run automatically and block bad changes. Here is how we integrate eval-gating into our CI/CD pipeline.

The CI Eval Stage

Our CI pipeline has a dedicated eval stage that runs after unit tests and integration tests but before deployment. The stage:

1. Checks out the golden dataset from a data registry (DVC or S3, depending on the project).
2. Runs the eval suite against the model produced by the build (or the configured API model).
3. Compares results to the baseline stored in the eval results database.
4. Fails the build if any metric crosses its regression threshold.

# ci-eval-stage.yml (simplified)
eval-job:
  runs-on: [self-hosted, gpu]
  steps:
    - uses: actions/checkout@v6
    - name: Pull golden dataset
      run: dvc pull eval-datasets/golden-v4
    - name: Run eval suite
      run: |
        python run_evals.py \
          --model ${{ github.sha }} \
          --dataset eval-datasets/golden-v4 \
          --output eval-results/${{ github.sha }}.json
    - name: Check regression thresholds
      run: |
        python check_regressions.py \
          --baseline eval-baselines/current.json \
          --candidate eval-results/${{ github.sha }}.json

Blocking vs. Non-Blocking

Not every regression has to block a deployment. We categorize metrics into three tiers:

This tiered approach prevents the eval gate from being too aggressive. Early on, we set every metric as blocking and the team stopped being able to deploy anything. Nobody benefitted from a gate that nobody could pass.

The Model Upgrade Problem in Depth

Model upgrades — whether switching providers, updating to a new version, or deploying a new fine-tune — are the highest-risk operation in an AI product. Here is our detailed process for making them safe.

Shadow Testing

Before any model switch, we run the candidate model in shadow mode alongside the production model. Shadow testing means the candidate sees the same production traffic but its outputs are discarded — never shown to users. We collect both outputs, log them to a comparison store, and analyze the differences.

# shadow_test.py — simplified structure
class ShadowTester:
    def __init__(self, production_model, candidate_model, sample_rate=0.05):
        self.prod = production_model
        self.candidate = candidate_model
        self.sample_rate = sample_rate
        
    async def handle_request(self, request, user_id):
        # Always serve from production
        prod_response = await self.prod.generate(request)
        
        # Maybe collect shadow data
        if hash(user_id) % 100 < self.sample_rate * 100:
            cand_response = await self.candidate.generate(request)
            await self.log_comparison(request, prod_response, cand_response)
        
        return prod_response

We run shadow testing for at least one week and at least 10,000 sampled comparisons before considering a switch. This gives us enough data to detect regressions that only appear on specific input distributions.

What Metrics to Compare in Shadow

The shadow comparison focuses on four categories:

Output quality. Automated metrics (semantic similarity, factual consistency against the source) plus LLM-as-judge evaluations comparing both outputs side by side. We flag any case where the candidate scores lower and the gap exceeds the metric’s noise threshold.

Behavioral differences. We categorize every shadow comparison into one of:

• Statistically identical (semantic similarity > 0.95, no structural change)
• Stylistic difference (similar information, different phrasing — acceptable)
• Content difference (different facts or claims — requires investigation)
• Format change (different output structure — may break downstream parsers)
• Refusal difference (one model refuses, the other answers — almost always a regression)

Latency and cost. The candidate might produce better outputs but take 3x longer or cost 10x more. These are regressions in the operational sense even if the quality is equal.

User-facing metrics correlation. If we have implicit feedback signals (user re-ran the query, clicked a different result, abandoned the session), we correlate those with the shadow comparisons. This is the closest we can get to A/B testing without actually serving the candidate.

# Deciding to switch — simplified decision logic
def should_switch(shadow_results):
    checks = {
        "quality_no_regression": 
            len(shadow_results.regressions) < len(shadow_results.improvements),
        "latency_acceptable": 
            shadow_results.candidate_p95_latency < PROD_P95_LATENCY * 1.2,
        "cost_acceptable": 
            shadow_results.candidate_cost_per_query < MAX_COST_PER_QUERY,
        "refusal_rate_safe": 
            shadow_results.candidate_refusal_rate <= PROD_REFUSAL_RATE * 1.1,
    }
    
    if all(checks.values()):
        return Action.SWITCH
    elif not checks["quality_no_regression"]:
        return Action.INVESTIGATE_REGRESSIONS
    elif not checks["refusal_rate_safe"]:
        return Action.ROLLBACK_CANDIDATE
    else:
        return Action.ESCALATE

How to Decide the Switch Is Safe

We use a decision matrix, not a single number. A model can be better on accuracy but worse on latency, and the right decision depends on your product’s priorities.

For a customer-facing chat product, we weight refusal rate and hallucination rate above all else — a model that answers more questions but hallucinates more is a net negative. For an internal summarization tool, we weight latency and cost more heavily because the alternative (no summary) is worse than an imperfect one.

The final decision always includes a staged rollout:

1. Canary (5% of traffic): Run for 24 hours, evaluate all metrics against production.
2. Expansion (25% of traffic): Run for 48 hours, watch user-facing metrics (engagement, retention, support tickets).
3. Full rollout (100% of traffic): Only after canary and expansion pass.

This staged approach means we can detect regressions on a small traffic slice before they impact the full user base. We have rolled back at the canary stage more times than we care to admit.

Building a Regression Test Suite from Production Logs

Your golden dataset is not enough. Production traffic always contains surprises — edge cases your curated set never captured. The best regression test suites are built from production logs.

Here is the process we follow:

Step 1: Collect raw logs. Every production request and response gets logged (with PII stripped). Include user feedback signals where available: thumbs up/down, ratings, re-requests, abandonment.

Step 2: Sample strategically. Random sampling is fine for monitoring, but for regression testing you want stratified sampling:

• High-frequency inputs (the most common 20% of requests) — ensures baseline performance is maintained.
• Edge-case inputs (rare formats, unusual lengths, borderline content) — ensures the model generalizes.
• Previously problematic inputs (past regressions, user-reported issues) — ensures fixed bugs stay fixed.
• Feedback-negative inputs (user downvoted or abandoned) — ensures the model isn’t regressing where it already struggles.

def sample_for_regression_suite(logs, n=500):
    high_freq = logs.top_p_requests(p=0.8).sample(n=n//2)
    edge_cases = logs.filter(is_edge_case).sample(n=n//4)
    regressions = logs.filter(was_regression).sample(min(len, n//8))
    feedback_neg = logs.filter(user_feedback == "negative").sample(min(len, n//8))
    return union(high_freq, edge_cases, regressions, feedback_neg)

Step 3: Auto-label. For regression purposes, the expected output is “what a good model would produce.” Since we cannot hand-label thousands of examples per week, we use the production model’s own output as the reference, then flag downstream feedback as a correction signal. If the production model produced output A and users rated it positively, output A becomes the expected reference for regression testing.

Step 4: Prune stale examples. Remove any example where the expected behavior has changed (e.g., product requirements shifted) or where the input is no longer valid (e.g., deprecated feature).

We refresh our regression suite from production logs every two weeks. It keeps the test set aligned with actual user behavior without requiring constant manual curation.

How Often to Run Regression Tests

The frequency depends on what you are testing and how fast your system changes.

The per-commit golden dataset eval runs only on a subset of the full suite — about 100 examples covering the most critical paths. The full nightly eval runs the entire suite (500-2000 examples) and generates a report that the team reviews the next morning.

When a production incident occurs, we add the failing case to the per-commit eval suite within 24 hours. This ensures the regression is caught on the next commit that touches the affected code path.

When a Regression Is Actually an Improvement

Here is the uncomfortable truth: your evaluation suite can be wrong. A model change that scores worse on your automated metrics might actually be better for users, and a change that scores better might be worse.

We have seen this happen multiple times:

• A summarization model that produced more concise summaries scored lower on ROUGE-L because it dropped more words. Users preferred the concise versions.
• A refactored prompt that reduced hallucination rate scored lower on “helpfulness” according to an LLM judge, which penalized the model for saying “I don’t know” instead of guessing.
• A safety filter upgrade that correctly refused harmful requests increased the refusal rate metric, triggering a false regression alert.

The solution is not to abandon automated evaluation — it is to treat your eval suite as a living artifact that you improve over time, not a fixed standard you worship.

When an eval gate blocks an upgrade that the team believes is beneficial, the correct response is:

1. Manually review the differences on a sample to confirm the team’s hypothesis.
2. Run a production shadow test to collect real user interaction data.
3. If user data confirms the upgrade is better, update the eval suite — either add new metrics that capture the improvement or adjust thresholds on existing metrics.
4. Document the incident so the next team hitting the same pattern knows what to do.

This feedback loop improves your eval suite over time. The first time a model upgrade gets blocked, you fix a single threshold. After a year of this process, your eval suite catches real regressions and rarely produces false positives.

Practical Takeaways

Here is what we have learned from building regression testing for AI products over the past two years:

Start with the golden dataset. Collect 200 curated examples with hand-labeled expected output characteristics. This is the single highest-leverage investment you can make for regression testing. It does not need to be perfect — it needs to exist.

Gate model upgrades, not just code changes. Prompt changes, model version bumps, and provider switches all need regression gates. CI should run evals on every commit that touches prompt files, eval configs, or model configuration.

Use shadow testing for high-risk changes. Never switch a model version directly from “old” to “new” without running both side-by-side on production traffic first. One week of shadow data is worth a month of offline eval tuning.

Calibrate your thresholds to noise. Zero-tolerance eval gates are counterproductive — they block every change and teach the team to work around the gate. Measure the variance of each metric and set thresholds at 2-3 standard deviations above noise.

Treat the eval suite as code. Version it, review changes to it, and update it when production behavior changes. Stale evals are worse than no evals because they create false confidence.

Build the feedback loop. Every false positive (eval blocked an improvement) and every false negative (eval passed but users saw regressions) is data that improves your eval suite. Log them, analyze them, and update your tests.

Track everything. Log every eval run, every model version, every prompt template, and every eval dataset version. MLflow’s experiment tracking is useful for comparing model versions side-by-side on your eval metrics, making it easier to detect when a model upgrade causes regressions on specific subsets of your data. When a regression eventually slips through (and it will), having the full history of eval runs lets you pinpoint exactly when and where the behavior changed.

Regression testing for AI products is not a solved problem. The tools are evolving, the models are changing, and every team’s failure modes are different. But the investment is worth it. The alternative — discovering regressions through support tickets — is not just expensive. It erodes user trust in ways that no metric can fully capture.