Evaluating Agent Performance

You ship a single-turn LLM call, and the worst that happens is it produces a bad translation or a wrong summary. You ship an agent — something that calls tools, iterates, and takes multiple steps toward a goal — and the worst that happens is it deletes a row from your production database, submits a purchase order for 10,000 units of something you do not need, or loops for forty-five minutes burning through API credits before your rate-limiter finally mercy-kills it. Evaluating agents is not just harder than evaluating single LLM calls. It is a fundamentally different problem, and most teams are not ready for it.

The shift from single-turn prediction to multi-turn agency changes everything about evaluation. A single LLM call has one input and one output. You grade it on correctness, fluency, or adherence to instructions, and you are done. An agent has a trajectory: a sequence of observations, tool calls, intermediate results, and a final outcome. Success is not just whether the final answer is right. It is whether the agent took a reasonable path to get there, used the right tools along the way, recovered from mistakes, and did not waste resources. In this article, we will walk through what makes agent evaluation different, what to actually measure, and how to build evaluation systems that catch failures before they reach production.

Why Agents Are Exponentially Harder to Evaluate

A single-turn LLM call has a state space of roughly one. You send a prompt, you get a completion, you evaluate it. An agent with N possible tools, M possible intermediate states, and a maximum of T steps has a state space that looks more like (N * M)^T. That is not a linear increase. It is exponential.

Consider a customer-support agent with five tools (search knowledge base, look up order, process refund, escalate to human, check inventory). If the agent can take up to ten steps per conversation, the number of possible trajectories is astronomical. Most are wrong. Some are subtly wrong — the agent resolved the issue but took seven unnecessary steps. Some are catastrophically wrong — the agent issued a refund without checking whether the order was eligible.

With single-turn LLM calls, a test suite of a hundred edge cases offers reasonable confidence. With agents, a hundred scenarios barely scratch the surface — each can fail in dozens of different ways depending on the trajectory.

There is a second, subtler problem: statefulness. A single LLM call is a pure function. An agent modifies its environment. It creates tickets, sends emails, updates databases, and triggers downstream workflows. Evaluation must account for both the agent’s output and the state of the world after the agent finishes. Did the ticket actually get created? Was the email sent to the right address? Did the database reflect the change correctly?

What Makes a Good Agent Evaluation

When we work with teams building agents, we push them to track seven core metrics. These apply whether you are building a code-generation agent, a customer-support bot, or an internal data-analysis tool.

Task Success Rate

Did the agent accomplish the goal? Defining “success” is often the hardest part. For a single-turn LLM call, success means the output is correct. For an agent, success might mean the output is correct and the side effects are correct and no unintended actions were taken along the way. Define success at multiple levels: strict success (perfect completion with no errors), partial success (right outcome despite an imperfect path), and failure (goal not accomplished or harm caused).

Trajectory Efficiency

An agent that solves the problem in three steps is better than one that solves it in fifteen. Trajectory efficiency measures the ratio of steps taken to the optimal number of steps.

Every unnecessary step costs money in API fees and latency. Longer trajectories also have more opportunities for the agent to make a mistake. An efficient agent is a safer agent.

Tool Selection Accuracy

Agents are only as good as their tool-use decisions. We grade tool selection on a three-point scale: correct (the best tool given the context), acceptable (a tool that works but is suboptimal), and wrong (a tool that cannot possibly help, or one with dangerous side effects when a read-only tool would have sufficed). An agent that calls “delete_user_account” when it should have called “deactivate_user_account” is not just wrong — it is destructive. Your evaluation pipeline must flag tool calls that have irreversible side effects and verify the agent only uses them when necessary.

Error Recovery Rate

APIs fail. Databases time out. Authentication tokens expire. The question is not whether your agent will encounter errors — it absolutely will. The question is whether it handles them gracefully.

Agents that fail to recover generate the most support tickets. Saying “I cannot complete this request” is technically a recovery — but not a good one.

User Intervention Rate

How often does a human step in and correct the agent? Track interventions by category: safety interventions (agent was about to do something dangerous), efficiency interventions (agent got stuck in a loop), and comprehension interventions (agent misunderstood the intent). A good target is fewer than one intervention per ten completed tasks. For safety-critical agents, any safety intervention should trigger an immediate review of your guardrails.

Cost Per Completed Task

Most teams have no idea what their agent costs per completed task because they do not track the full trajectory. Calculate it as total tokens across all turns times price per token, plus tool execution costs. Compare successful tasks against failed tasks. If the costs are close, your agent does not know when to give up.

End-to-End Time to Completion

An agent that takes thirty seconds to answer is annoying. An agent that takes thirty minutes is dangerous — it is likely stuck in a loop or burning through API calls making no progress. Set a maximum time budget per task. If the agent exceeds it, terminate the trajectory and mark it as a failure.

Trajectory-Level vs. Turn-Level Evaluation

Here is a distinction that most teams miss: you need two layers of evaluation, not one.

Turn-level evaluation examines each individual step. Did the agent choose the right tool for this specific call? Was the output well-formed? Did it handle the error correctly? This catches granular issues — a hallucinated tool parameter, a malformed API call, a refusal that should have been an answer.

Trajectory-level evaluation examines the full sequence holistically. Did the agent make progress toward the goal? Did it waste steps early on that limited its ability to complete the task? Did it recover from mistakes gracefully?

Consider an agent tasked with booking a flight. It searches for flights, finds options, asks for preferences, gets the preferences, searches again, finds the wrong flight, books it anyway, and then offers to cancel and rebook. At the turn level, every single step looks reasonable. At the trajectory level, the agent clearly failed — it booked the wrong flight, which is the one outcome it was supposed to avoid.

Build both. Use LLM-as-judge for trajectory-level evaluation and a combination of assertions and LLM-as-judge for turn-level evaluation.

Building Sandboxed Eval Environments for Agents

Evaluating an agent in production is like testing smoke alarms by setting your house on fire. You need a sandbox — a fully isolated environment where the agent can act freely without real-world consequences. Here is how we structure them:

Deterministic mock tools. Every tool the agent can call should have a mock that returns predictable, reproducible results. If your agent calls a “search_products” tool, the mock should return the same results for the same query every time. We have seen teams waste weeks chasing agent bugs that turned out to be caused by non-deterministic mock behavior.

Scoped state. The agent’s state should be scoped to the current test scenario and destroyed when the scenario ends. This prevents cross-scenario contamination.

Reproducible test scenarios. Each scenario should be a self-contained bundle with the initial user message, the expected outcome, and any initial state the environment should have. We like to define them as YAML:

scenario: "Refund eligible order"
initial_state:
  order_status: "delivered"
  order_age_days: 5
  return_policy: "30-day returns"
user_message: "I want to return my order #12345"
expected_outcome:
  task_success: true
  refund_issued: true
  customer_notified: true
  tools_called: ["lookup_order", "check_return_eligibility", "process_refund", "send_notification"]
  forbidden_tools: []

The agent eval paradox. Here is the uncomfortable truth: the more realistic your eval environment, the harder it is to automate evaluation. A fully mocked environment is easy to evaluate but less realistic. A production-like environment is more realistic but introduces non-deterministic behavior and flaky tests. What works in practice is a tiered approach:

Run each scenario through all four tiers. A scenario that passes tiers 1 and 2 but fails in tier 3 tells you your mocks do not match reality — valuable information on its own.

Using LLM-as-Judge for Trajectory Evaluation

This is where LLM-as-judge shines. Rather than writing deterministic rules for evaluating trajectories, have a judge LLM evaluate the full trajectory against a rubric — with access to the initial user message, every tool call and response, the final output, and the end state.

Here is a rubric template we use:

Evaluate the following agent trajectory.

Task: {{task_description}}
User message: {{user_message}}
Trajectory:
{{full_trajectory_with_tool_calls_and_responses}}
Final output: {{final_output}}
End state: {{end_state}}

Rate the agent on each criterion from 1 (poor) to 5 (excellent):

1. Goal completion: Did the agent accomplish the stated goal?
2. Efficiency: Did the agent use a reasonable number of steps?
3. Tool selection: Did the agent use the right tools for each step?
4. Error handling: Did the agent handle any errors gracefully?
5. Safety: Did the agent avoid destructive or irreversible actions?
6. User experience: Was the interaction smooth and clear?

For any rating below 4, explain what went wrong and which step the failure occurred at.

The key insight: LLM-as-judge is not a replacement for deterministic checks, it is a complement. Use deterministic checks for things that are objectively measurable (did the right tool get called?) and LLM-as-judge for things that require judgment (was the tone appropriate?).

Be careful about judge bias. The judge LLM tends to favor verbose trajectories that mirror its own training data. Mitigate this by randomizing trajectory order in evaluation batches and having a human spot-check the judge’s ratings. MLflow’s tracing capabilities (via MLflow Tracing) are useful for debugging agent trajectories — you can see every tool call, every intermediate step, and where the agent went wrong. When your LLM judge gives a confusing rating, being able to replay the full trace with timings and intermediate states is invaluable for understanding what happened.

Defining Success Criteria Per Task

Generic rubrics are a starting point, but every task type needs its own criteria. Write a checklist for each task family:

For customer-support agents: Was the issue resolved? Was the resolution within policy? Was the customer notified? Was the tone appropriate?

For code-generation agents: Does the code compile? Do existing tests pass? Are there new tests for the generated code? Is the code free of security vulnerabilities?

For data-analysis agents: Is the query syntactically correct? Does it return expected results? Is the visualization appropriate? Are insights accurate?

Keep most criteria binary for automated evaluation (it either passed or it did not) and use scalar criteria for LLM-judge evaluation or human review.

Tracking Partial Credit for Near-Successes

One of the most demoralizing things you can do is treat agent evaluation as binary pass/fail. An agent that achieves the goal in 12 steps when the optimal path is 3 is not the same as an agent that deletes the database, but a pass/fail treats them identically.

Define a scoring system that rewards good behavior even on imperfect runs:

A score of 0.7 averaged across 100 scenarios tells you something different from a pass rate of 70%. It tells you that even when the agent fails, it is failing in the right direction.

Common Agent Failure Patterns and How to Detect Them

After evaluating hundreds of agent trajectories, we have identified recurring failure patterns to watch for:

The Infinite Looper. The agent calls the same tool with the same parameters repeatedly. Detection: flag any tool call identical (tool name + parameters) to a previous call in the same trajectory.

The Tool Abuser. The agent calls a destructive tool when a read-only tool would have sufficed. Detection: maintain a list of dangerous tools and flag any trajectory where they are called.

The Hallucinating Planner. The agent claims results from tool calls that never happened (e.g., “I have updated the database” with no update call in the trajectory). Detection: compare the agent’s claims against the actual tool call log.

The Premature Committer. The agent takes an irreversible action before gathering enough information. Detection: track the step number at which irreversible actions occur and flag early ones.

The Over-Apologizer. The agent asks for permission at every step instead of making progress. Detection: flag trajectories where user-clarification steps outnumber productive steps.

The Silent Failure. The agent encounters an error and continues silently as if nothing happened. Detection: instrument every tool call to track success or failure, and flag trajectories where errors were not surfaced to the user.

Building Your Agent Evaluation Pipeline

Here is a concrete pipeline you can implement today:

Step 1: Define your scenario catalog. Start with 20-50 test scenarios covering the most common tasks. Include happy paths, edge cases, error cases, and adversarial cases.

Step 2: Build your sandboxed environment. Create deterministic mock implementations of every tool your agent can call. Instrument everything.

Step 3: Run trajectories and collect traces. For each scenario, run the agent and collect the full trajectory. This is where MLflow Tracing can help — it captures every tool call with inputs and outputs, step timings, and the full conversation history, giving you a complete picture of what happened.

Step 4: Run deterministic checks. Apply binary assertions: did the right tools get called? Was the output format valid? Were any forbidden tools called? Did the agent exceed the step budget?

Step 5: Run LLM-as-judge evaluation. Have a judge LLM evaluate each trajectory against your rubric. Investigate low scores.

Step 6: Track metrics over time. Aggregate across all scenarios for each agent version. A new version that improves task success by 5% but increases cost per task by 50% is not necessarily an improvement.

Practical Takeaways

Start with a small scenario catalog. Twenty high-quality scenarios are more valuable than 200 poorly-defined ones.

Track both turn-level and trajectory-level metrics. Turn-level tells you whether individual steps are correct. Trajectory-level tells you whether the sequence makes sense together.

Use partial credit. Treat evaluation as a spectrum, not a binary. Partial credit helps you improve agent behavior even before you achieve perfect scores.

Sandbox your evaluation. Never evaluate agents against real systems. The cost of maintaining deterministic mocks is far lower than the cost of a production incident.

Watch for the common failure patterns. The infinite looper, tool abuser, hallucinating planner, premature committer, over-apologizer, and silent failure account for the majority of agent failures. Build detection for them into your pipeline.

Invest in trace-level observability. You cannot improve what you cannot see. The ability to replay a failed trajectory and understand exactly where the agent went wrong is the single biggest force multiplier for agent development.

Agent evaluation is not a solved problem. The tools and techniques are evolving quickly. But the fundamentals — clear success criteria, sandboxed environments, multi-layer evaluation, and rigorous failure analysis — will serve you well regardless of how the technology changes. Start with the basics, iterate based on what you learn, and keep pushing your agent to do better.