Building Continuous Evaluation Systems

We noticed something unsettling about six months into running our first production AI system. The quarterly evaluation scorecard showed consistent 88-92% accuracy. Users were not complaining. Yet conversation completion rates were slowly ticking down. Support ticket deflection was eroding. Task success rates in A/B experiments were drifting. The quarterly eval was telling us everything was fine while the product was quietly getting worse.

We had built the evaluation equivalent of checking a patient’s temperature once a season. And we see this pattern everywhere: teams point to a static score against a frozen test set as if it captures the health of a living system. It does not. The model changes when you fine-tune, swap providers, or tweak a prompt. Your data changes as user behavior shifts, new features launch, and seasons turn. The definition of “correct” changes — an answer that passed muster in January is unacceptable by June. Point-in-time evaluation tells you what your system did on one afternoon. It tells you nothing about what it is doing right now.

This article is about the alternative: continuous evaluation. Not a process or a dashboard, but an architectural layer that sits beside your AI system and constantly answers, “Are we getting better or worse?”

The Case Against Point-in-Time Evaluation

Three independent sources of drift invalidate any point-in-time evaluation.

Model drift. You deploy a new version of your system — a cheaper model, a tweaked system prompt, a different chunking strategy. The eval you ran three weeks ago against the old system tells you nothing about the new one. We have watched teams ship model changes that dropped effective accuracy by 12 points, discovered only weeks later because the next scheduled eval had not run yet.

Data drift. Your users change what they ask about. In January, onboarding flows. In March, billing edge cases. In June, a new product launch shifts the distribution entirely. A test set built from January traffic now measures how well your system handles a distribution that no longer exists. We worked with a team whose eval scores actually improved over three months — not because the system got better, but because the dataset drifted toward easier questions as it aged out of sync with production.

Concept drift. The ground truth changes. A response factually correct in Q1 might be misleading by Q2 due to a policy change, product update, or new information. A chatbot that tells users the old refund policy is “correct” by the training data and “wrong” by the world. Continuous evaluation with fresh ground truth catches this. Quarterly evals miss it entirely.

The math is simple: if any of these changes between evaluations, your last score is a historical artifact, not current measurement. And all three change faster than most teams evaluate.

Architecture of a Continuous Evaluation System

A continuous evaluation system has six layers. Each is independently useful; together they catch regressions in minutes rather than quarters.

Layer 1: Data Collection

You need a stream of inputs to evaluate. There are three sources, and you need all three.

Production logs give you what is actually happening. Synthetic data covers what could happen. Curated datasets provide ground truth for critical scenarios. We recommend starting with a sampling layer at the API gateway that captures every request with a configurable sampling rate, storing raw input, output, and response metadata (latency, token count, model used, temperature) in Parquet files on object storage. Do not evaluate everything at this stage — just capture it.

Layer 2: Evaluation Triggers

Not all evaluations need to run all the time. Four trigger patterns cover the vast majority of use cases.

Commit-triggered evaluation. Every push to evaluation config, system prompt, model selection, or retrieval pipeline triggers a full evaluation suite and blocks deployment if scores drop below threshold. This is the eval equivalent of CI/CD and the single highest-leverage investment you can make.

Scheduled evaluation. Runs a representative subset on a cadence: hourly for smoke tests (latency, refusal rate, format compliance), daily for quality scoring (accuracy, relevance, faithfulness), weekly for deep dives (adversarial robustness, bias audits). Match the schedule to your risk tolerance — if an hour-long undetected regression costs you, run hourly.

Drift-triggered evaluation. When a drift detection layer flags a statistically significant shift in input distribution, output distribution, or behavioral metrics, the system runs a focused suite against recent production traces. This catches the cases where no one pushed a change, but the world changed beneath you.

On-demand evaluation. A human triggers a full eval run at any time. Critical for incident response — when a user reports a quality issue, you need to determine whether it is systematic or a one-off within minutes.

Layer 3: Evaluator Execution

This is the engine. It takes inputs, applies evaluators, and produces scores. Two architectural principles matter: parallelism and caching.

Evaluators should be stateless and idempotent. Each receives an input, optionally an expected output, and returns scores. This lets you run them in parallel across a pool of workers. A single machine running background processes suffices for months. Eventually you will want a job queue (Celery, RabbitMQ, or SQS) to scale workers independently.

Cache aggressively. If the same input appears in production logs and a synthetic suite, evaluate it once. If a prompt change only affects one evaluator, cache results for unaffected evaluators. We use a content-addressed cache keyed on (input, system_config_hash, evaluator_name) with a TTL matching the eval schedule. This cut our compute costs by roughly 60%.

Be thoughtful about evaluator frequency:

Start with format and behavioral checks — they catch the most common regressions at the lowest cost. Add quality scorers once the pipeline works. Add LLM-as-judge only for dimensions that quality scorers miss. Add human review for calibration.

Layer 4: Results Aggregation

Individual scores are noise. Aggregated scores are signal. Store results in a time-series database (ClickHouse, TimescaleDB, or well-structured PostgreSQL) keyed on (eval_run_id, evaluator_name, model_version, timestamp, input_hash). The aggregation layer computes:

• Per-evaluation scores: Mean, median, and distribution for a single run. Used for pass/fail decisions.
• Trends: Scores over sliding windows — last hour, day, week, 30 days. This is where you see drift before it becomes a crisis.
• Comparisons to baseline: Current score vs. previous run, previous day’s average, score at last deployment. Compute absolute and relative deltas with statistical significance — a 2-point drop on 10,000 samples differs from a 2-point drop on 10 samples.
• Scores by segment: Breakdown by input category, user cohort, model version. This is how you discover the system handles billing fine but fails on international shipping queries.

The aggregation layer should produce two outputs: a materialized view for dashboards and a stream of events for alerting.

Layer 5: Alerting and Gating

Continuous evaluation without automated response is an expensive dashboard.

Alerting. A practical hierarchy:

Tune thresholds using historical data. Set critical threshold at two standard deviations below the 30-day mean. Set warning at one standard deviation. Avoid paging people for normal variance.

Gating. For commit-triggered evals, a failing evaluator should block deployment — the equivalent of a failing CI test. Use a “gating scoreboard”: a set of evaluators that must all pass at defined thresholds. A change that drops accuracy by more than 3 points or increases refusal rate by more than 1 point? Blocked. Investigate and either fix the regression or update thresholds with a documented rationale. A properly tuned gating system has a false positive rate under 2%.

Layer 6: Drift Detection

This connects evaluation to observability. Monitor three things:

• Input distribution drift. Track embeddings of incoming queries and compare to a 7-day reference window. When the distribution shifts significantly (using population stability index or Maximum Mean Discrepancy), trigger an evaluation. This catches data drift before it impacts quality.
• Output distribution drift. Track output embeddings the same way. Output drift often precedes quality degradation — the model changes phrasing and structure before it gets things wrong.
• Behavioral metric drift. Track refusal rates, output length, response time, token usage. A sudden change in any is a leading indicator of trouble.

Drift detection runs on every request (or a high-frequency sample) and writes to a time-series store. When it flags a shift, it triggers evaluation against the actual traffic that set off the alert — not against the synthetic test set.

Online vs. Offline Evaluation

Continuous evaluation comes in two flavors, and you need both.

Online evaluation runs on individual production requests in real time. It answers: “Was this response good or bad?” immediately. It is limited to what can be measured without a reference answer — format checks, refusal detection, output length, tone analysis, latency. The challenge: it cannot assess correctness or faithfulness without ground truth, and it is constrained by latency and cost (you cannot run an LLM-as-judge on every request without breaking your budget or p95).

Offline evaluation runs asynchronously in batches against stored production traces. This is where you run quality scorers, LLM-as-judge evals, human reviews, and baseline comparisons. It gives you depth. It catches the regressions online checks miss.

The recommended architecture: online evaluators run on every request as a fast pre-flight check and log results to the same time-series store. If an online evaluator flags an anomaly — unexpected output format, refusal when none expected, high toxicity — it triggers an immediate offline evaluation of that specific request and a sample of similar recent ones. Real-time detection with deep-dive verification.

Sampling Strategies for Continuous Eval at Scale

You cannot evaluate every request with your full suite. Use stratified sampling: partition requests by input category, model version, user tier, or other meaningful segments. Sample a configurable number from each strata on a rolling basis.

A practical starting point: 100 requests per hour per category. Five categories equals 500 evaluations per hour — 12,000 per day. Manageable for most systems, and gives statistically significant results within hours of a regression.

Sample more aggressively for high-risk categories (medical advice, financial calculations, legal interpretations). Sample just enough to catch format-level regressions for low-risk categories (weather queries).

Adaptive sampling reduces cost further: increase the sampling rate when drift is detected, decrease it during stable periods. The drift detection layer feeds back into the sampling decision.

Storing Eval Results for Trend Analysis

Raw scores are point-in-time measurements. The trend over time is what makes them valuable.

Store every result with an immutable timestamp and version identifiers (system config hash, evaluator version, model version). This creates a historical record that answers: “Was our system more accurate in April?” and “Did the prompt change on May 15 degrade faithfulness?”

The schema:

CREATE TABLE eval_results (
    id UUID PRIMARY KEY,
    timestamp TIMESTAMPTZ NOT NULL,
    input_hash TEXT NOT NULL,
    system_config_hash TEXT NOT NULL,
    evaluator_name TEXT NOT NULL,
    model_version TEXT NOT NULL,
    score FLOAT NOT NULL,
    score_metadata JSONB,
    trigger_type TEXT,
    trigger_context TEXT
);

CREATE INDEX idx_eval_results_timestamp ON eval_results (timestamp);
CREATE INDEX idx_eval_results_evaluator ON eval_results (evaluator_name, timestamp);

With this data, you can compute trends over any window, compare any two configurations, and identify when regressions started — not just when someone noticed them.

The single most useful view: scores over the last 30 days with a 7-day rolling average. It smooths daily variance and shows actual trajectory. When that 7-day average crosses below a threshold, you have a problem worth investigating.

Eval Freshness

Your evaluation suite itself ages. The synthetic test data from January might not cover features shipped in March. The curated dataset for v1 might include invalid examples. Behavioral checks configured for the original system prompt might miss the failure modes of the new one.

This is eval freshness, and it is one of the most overlooked aspects of continuous evaluation. Your evals need maintenance, versioning, and regular review.

Run an eval audit on a monthly cycle. For each evaluator:

• Are the test inputs still representative of production traffic?
• Are expected outputs still correct given current policies and knowledge?
• Is the threshold still appropriate, or has the system improved to the point where it is trivially easy?
• Are there new failure modes no evaluator covers?

An eval that never changes is an eval that is slowly lying to you.

Tools and Infrastructure

Here is a pragmatic roadmap for teams at each stage.

Week 1 — If you have nothing. Add a logging middleware to your API gateway that writes every request and response to Parquet files in object storage. This takes an afternoon and gives you the raw material for everything else. Then write a single Python script that reads the last hour of logs, runs format and refusal checks, and writes results to SQLite or a Google Sheet. Run it on cron. Not scalable, but it will catch regressions your current process misses entirely.

Week 2 — Add eval triggers. Set up a CI workflow that runs your eval script against held-back production traces on every prompt change. Make it a required CI check. Start with 50-100 eval examples. This creates the gating mechanism.

Month 1 — Build the pipeline. Replace cron with a proper orchestration tool — Airflow, Prefect, Dagster, or even GitHub Actions. Add more evaluators: relevance, faithfulness, output format validation. Build an aggregation layer. Set up a simple dashboard (Grafana on PostgreSQL works well). Configure alerting for critical thresholds. By the end of month one, evaluations run automatically, results are stored historically, and someone gets paged when quality drops.

Quarter 1 — Add drift detection and adaptive sampling. Implement input and output embedding tracking with a lightweight model. Set up drift detection using population stability index. Wire it to trigger evaluations. Implement stratified sampling. Your evaluation system is now continuous and self-correcting.

Beyond. Add LLM-as-judge evaluators for dimensions automated scorers cannot cover. Set up human review rotations. Build eval regression test suites that reproduce every incident as a test case. The system is never finished — it grows with your understanding of your quality dimensions.

A Note on Culture

The hardest part of continuous evaluation is not the technology. Evaluation results are uncomfortable. They surface problems teams have been ignoring or have not noticed. They make visible the gap between belief and reality.

The teams that succeed treat eval scores as a tool for improvement, not a report card. They celebrate when the system catches a regression, even if it blocks a launch. They invest in eval infrastructure as a first-class engineering concern, not a side project. They accept the evaluation suite will never be perfect and iterate on it alongside the product.

If you take nothing else from this article, take this: the purpose of evaluation is not to prove your system is good. The purpose is to find out how to make it better. Continuous evaluation, done right, is the tightest feedback loop you can build between your system’s behavior and your team’s understanding of that behavior. Start with production logs and a cron job this week. Add gating next week. Build the pipeline this month. The alternative — quarterly reviews against a frozen test set — is not evaluation. It is archaeology.