Building an Evaluation Pipeline

We shipped a feature to production that, for about one user in twenty, started replying in French. The model hadn’t changed. The prompt hadn’t changed. What changed was a dependency update that subtly shifted how a preprocessing function handled accented characters, and our manual testing session — two engineers running five hand-picked examples — simply didn’t catch it. We learned the hard way that “testing feels fine” is not a signal. An evaluation pipeline is what turns “feels fine” into “here is the number, and it went down by 12%.” If you are building anything on top of a language model and you do not have one yet, this article is for you.

What an Eval Pipeline Actually Is

An evaluation pipeline is an automated system that runs a fixed set of inputs through your application, scores the outputs against predefined criteria, and surfaces whether those scores changed. It is the testing harness for the part of your stack that cannot be tested with assertions like assert response.status_code == 200.

A pipeline has three stages:

1. Run — Send your eval dataset through the application or model under test.
2. Score — Apply evaluators (heuristic checks, model-as-judge, human labels) to each input-output pair.
3. Gate — Compare aggregate metrics against thresholds and decide whether to proceed.

That is it. The complexity lives in the details of each stage, and the sophistication of a pipeline tends to scale with how bad the consequences of a bad deployment would be. A customer-facing chatbot handling billing questions needs a pipeline that catches subtle tone shifts. An internal summarization tool that nobody reads carefully might get away with a single sentence-level ROUGE score.

The critical insight is that an evaluation pipeline is not a one-time benchmarking exercise. It is a regression detection system. You run it continuously — on every PR, every model update, every prompt change — and you care most about the delta, not the absolute number.

Step 1: Build Your Eval Dataset

The eval dataset, often called a golden dataset, is the foundation of everything. Get this wrong and no amount of fancy evaluators will save you.

What Makes a Good Golden Dataset

A good eval dataset has three properties:

For a classification system (moderation, routing, intent detection) you want roughly 200–500 labeled examples per class, balanced or weighted to match production distribution. For a generation system (summarization, Q&A, email drafting) you want 100–300 examples where each one targets a specific capability: handling long context, following formatting instructions, not hallucinating facts.

Real example. We built a support-ticket triage system. Our first golden set had 50 examples — 40 “billing” and 10 “technical.” When we ran evals, scores looked great. In production, the system routed 30% of technical tickets to billing. The eval had zero coverage of edge cases like “the user mentions both payment and an error.” We rebuilt the set with 300 examples distributed across 6 categories, plus 50 deliberately ambiguous edge cases. The eval started catching real regressions immediately.

How to Build One

Start from production logs, not from your imagination. Pull the last 500–1000 real inputs, cluster them by output type or difficulty, and sample 10–20 from each cluster. For each example, define the signal you will measure:

• Classification: the expected label (hard ground truth).
• Generation: criteria the output must satisfy, phrased as statements a judge can verify: “the summary includes the refund amount,” “the tone is professional,” “no invented names appear.”

For generation tasks, a single label is rarely enough. Each example should have 3–5 criteria statements. This lets you compute pass rates per criterion and identify what is degrading when a score drops — is the model forgetting facts, or just getting less polite?

Avoid these pitfalls:

• Leaking into training data. If your eval set overlaps with training data, improving scores may just measure memorization. Hash each input and check against a holdout set.
• Stale distribution. Re-sample from production every 4–6 weeks. Distribution shifts — new product lines, new user behaviors — silently invalidate old eval sets.
• Too few hard examples. Easy examples never fail, which means your eval never fires. Deliberately include edge cases: ambiguous queries, very long inputs, multilingual text, adversarial prompts.

Step 2: Structure Your Evaluators

An evaluator is a function that takes an input and an output and returns a score — usually 0 or 1 for pass/fail, or 1–5 for a rubric score. We see three tiers of evaluator in practice.

Heuristic Evaluators

These are deterministic checks: no LLM involved, no API call, sub-millisecond per example.

def contains_required_sections(output, required_sections):
    for section in required_sections:
        if section.lower() not in output.lower():
            return 0
    return 1

def response_length_in_bounds(output, min_words=50, max_words=500):
    word_count = len(output.split())
    return 1 if min_words <= word_count <= max_words else 0

def no_prohibited_phrases(output, blocked_list):
    for phrase in blocked_list:
        if phrase.lower() in output.lower():
            return 0
    return 1

Heuristic evaluators are cheap, fast, and never hallucinate. Use them for everything they can handle: format compliance, length constraints, banned terms, regex patterns, JSON schema validation. They should cover 40–60% of your evaluation surface for free.

Model-Based Evaluators (LLM-as-Judge)

For qualities that cannot be reduced to a string check — tone, helpfulness, factual accuracy — you need a model-based evaluator. This means prompting a second model to score the output of the first.

The cardinal rule: your evaluator model must be at least as capable as the model being evaluated. Using a small, cheap model to judge a frontier model produces noisy, biased scores. We see teams burn weeks chasing regressions that were actually evaluator noise.

A good evaluator prompt follows this structure:

You are evaluating a response. Here is the user query:
{input}

Here is the response:
{output}

Score the following criteria as PASS (1) or FAIL (0):

1. Factual accuracy: Does the response contain any
   statements not supported by the provided context?
2. Completeness: Does the response address all parts
   of the user query?
3. Tone: Is the response professional and courteous?

Return a JSON object:
{"factual_accuracy": 0|1, "completeness": 0|1, "tone": 0|1}

Run each example through the evaluator model with temperature 0. If you can afford it, make 3 evaluator calls per example and take the majority vote — this smooths out the judge’s own inconsistency.

Cost reality. Evaluating 200 examples with GPT-5.4 as judge costs roughly $0.50–$1.50 per run, depending on output length. On every commit, that is trivial. On every token generated in an interactive session, it adds up. Batch your evaluations and run them on CI, not on every keystroke.

Human Evaluation

Human evaluation is the gold standard and the most expensive signal per datapoint. Use it sparingly and strategically:

• Calibration. After you tune a model-based evaluator, run 50–100 human evaluations to check that the LLM judge agrees with human raters. Measure Cohen’s kappa or simple agreement rate. If agreement is below 80%, your judge prompt needs work.
• Adversarial discovery. Every 2–4 weeks, have a human review 50–100 production outputs looking for failure modes the automated evals missed. Then add those as golden examples.
• Boundary cases. When an automated eval flags a borderline score change, route those examples to human review before making a deploy decision.

Step 3: Integrate Evals Into CI

An eval pipeline that runs once a month on a developer’s laptop is not a pipeline — it is a ceremony. A pipeline runs on every PR and surfaces results in the review workflow.

Here is the integration pattern that works for teams of 2–25 engineers:

1. Trigger. On every PR that changes a model config, prompt template, or inference code, run the eval suite. For teams with heavy CI usage, cache eval results for identical inputs and only re-run affected subsets.

2. Parallelize. Each example is independent. Run them in parallel batches. With 200 examples and an evaluator that takes 500ms per call, a serial run takes 100 seconds. With 10 parallel workers, it takes 10 seconds. Most teams can achieve this with a simple job matrix or a lightweight orchestration library.

3. Report. Output two numbers: the aggregate score (overall pass rate across all examples) and the delta from the main branch baseline. A CI comment like “Eval pass rate: 94.2% (-1.3pp vs main)” is actionable. A wall of 200 individual scores is noise.

4. Fail the build. Set a threshold and gate merges on it. More on that next.

Practical CI Integration

# .github/workflows/eval.yml — minimal pipeline
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'app/**'
      - 'configs/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - name: Run evaluation pipeline
        run: python run_eval.py
      - name: Check thresholds
        run: python check_thresholds.py

We use a single run_eval.py that loads the golden set, runs inference, applies evaluators, and writes results to a JSON file. The check_thresholds.py reads the results, compares to thresholds defined in a config file, and exits non-zero if any threshold is breached. The non-zero exit fails the CI check.

Step 4: Choose Metrics That Match Your Use Case

One metric does not fit all. Here is what we see working in practice for common task categories.

Classification Metrics

For classification, also track a confusion matrix per class. A drop from 98% to 97% accuracy sounds fine until you notice that class 4 went from 85% to 45%. Aggregates lie.

Generation Metrics

The most useful single number for generation tasks is the pass rate across all criteria and all examples. It is coarse but directionally reliable. When it drops, you know something is wrong — then you drill into which criteria and which examples failed.

RAG Metrics

Retrieval-augmented generation adds a retrieval step that can fail independently of generation.

Track retrieval metrics and generation metrics separately. A drop in generation quality could be a retrieval problem (bad context fed to the model) or a generation problem (model ignoring good context). Separate metrics tell you which.

Step 5: Set Thresholds and Gate PRs

Thresholds are the most politically charged part of an eval pipeline. Engineers hate being blocked by a flaky score, and product managers hate deploying broken features. Find the middle ground.

How to Set Initial Thresholds

Do not guess. Run the eval on the last 10 commits that deployed without incident, plus 3–5 that caused known problems. The thresholds live between these two distributions.

Soft vs Hard Gates

A hard gate fails CI when a threshold is breached. Use this for things that are clearly bad: the model stopped producing valid JSON, the response length doubled, toxic content rate went above 0.5%.

A soft gate passes CI but flags the PR for review with a warning banner. Use this for metrics that are noisy or still stabilizing: helpfulness score dropped 4pp but the sample is small.

In practice, we see teams converge on 1–2 hard gates (format compliance, safety) and 3–5 soft gates (quality metrics). The hard gates prevent catastrophic deploys. The soft gates build a culture of quality without creating workflow friction.

How Eval Drift Happens and How to Stop It

Eval drift is the silent killer of evaluation pipelines. It happens when the eval dataset, the evaluators, or both stop measuring what actually matters.

Dataset Drift

Six months after launch, your golden set still looks like the first 500 production examples. But your production distribution has shifted — new user segments, new query types, new content formats. The eval says 97% pass rate. Production is a mess.

Fix. Every 4–6 weeks, sample 50–100 new production inputs, manually label them, and swap out the oldest 10–20% of your golden set. Track the per-example pass rate over time. If some examples have passed for 50 consecutive runs, they are likely too easy. Replace them.

Evaluator Drift

Your model-based evaluator prompt was tuned for GPT-4o. Now you are evaluating Claude outputs. But the evaluator prompt has implicit assumptions about writing style, verbosity, and structure that penalize the new model unfairly. The eval score drops, but the actual quality is fine.

Fix. Every time you change the model under evaluation, re-run 50 human evaluations on the same set and check agreement with your automated evaluators. If agreement drops below 75%, retune the evaluator prompts. Better yet, use an evaluator model from a different provider than the model being evaluated, to avoid same-model bias.

Threshold Decay

Thresholds set six months ago looked reasonable then. But as the system improves, every run scores 97–99%. The thresholds never fire, so nobody reads the eval reports. Then a change drops the score to 92%, which is still above the 90% threshold, and ships to production with visible quality loss.

Fix. Review thresholds quarterly. Raise them as the system improves. The eval should fire occasionally — if it never fires, the threshold is useless.

A Minimal Viable Pipeline for a Small Team

If you are on a team of 2–3 engineers and you have no eval pipeline today, here is what “good enough” looks like. You can build this in a week.

The whole pipeline, including the CI config, should be under 500 lines of code. Do not over-engineer it. A simple pipeline you run every day is infinitely more valuable than a perfect pipeline you run once a quarter.

Practical Takeaways

• Start with the eval dataset, not the evaluators. A mediocre evaluator on a well-constructed dataset catches more regressions than a brilliant evaluator on garbage data. Spend 60% of your eval build time on the dataset.

• Heuristics are free, use them first. Before writing a single model-based evaluator, ask: can I check this with a regex, a length constraint, or a schema validator? Most format and safety requirements can be. Reserve LLM-as-judge for things that genuinely require judgment.

• Run on every PR, not on deploy. The entire point is catching regressions before they reach production. Running evals only during deployment means you are measuring the problem, not preventing it.

• Track deltas, not absolutes. A 92% pass rate is meaningless in isolation. A 92% pass rate that dropped from 94% is a signal. Store baselines. Compare against them. Fail on regressions.

• The pipeline is a living thing. Eval datasets atrophy. Evaluator prompts go stale. Thresholds become irrelevant. Schedule a quarterly review of the entire pipeline. If you skipped it last quarter, the pipeline is already drifting.

The French-speaking incident cost us a day of rollback, a hotfix, and an apology post. The eval pipeline we built afterward has caught nine regressions in five months — subtle things like dropped context windows, formatting breakage, and a prompt injection bypass that a human tester never would have found on a Friday afternoon. It is the single highest-leverage investment we have made in our AI system’s reliability. And it runs in under two minutes on every pull request.