Agent evaluation: What to measure, how and when

Today, LLM agents are everywhere and they are fundamentally changing the way we work. They are revolutionizing automation, decision support, and enterprise workflows. Companies are integrating agents into their ecosystems while more platforms are offering agents as a core offering. The buzz is warranted; agents bring a level of power that attracts tech and non-tech crowds alike. Building an agent and observing its capabilities creates excitement. You can see the agent take instructions, interact with its environment, and take actions to achieve goals. That’s great. But how do we know if the agent is doing a good job? Let’s say we make changes to our agent; do we know the effect of those changes? This is where evaluations come into play.

Evaluations exist throughout the entire agent development lifecycle. We focus on testing and validation during the development phase, then shift our attention to monitoring once the agent is in production. 2025 was dubbed the year of agent POCs. Now, companies are ready to take agents into production. Evaluation will allow teams to ship agents with confidence and build trust.

Understanding agent evaluation

What are agents?

Before diving into how to evaluate an agent, let’s define it.

Agents are autonomous systems generally powered by a large language model (LLM) and capable of taking actions required to achieve objectives across multiple steps. Unlike traditional software that generally maps inputs to outputs, agents have the liberty of reasoning and deciding what action to take. Powerful but more difficult to control.

An agent is generally made up of three key components:

• Reasoning engine: typically an LLM that analyzes information and makes decisions on what actions to take.

• Tools: external capabilities the agent can invoke like an API call or web search. The sequence of actions and tool calls is known as a trajectory and is a major focus of agent evaluations.

• Memory: how the agent stores and retrieves context such as conversation history and user preferences.

What is agent evaluation?

Agent evaluation is a structured process of measuring an agent’s behavior. We assess how well the AI system can autonomously perform tasks, leverage tools to take actions, and reason over multiple steps to achieve an objective.

For practitioners experienced with evaluating traditional machine learning systems, agent evaluation introduces a more comprehensive set of evaluations. Traditional ML evaluations focus primarily on a model’s performance on a fixed dataset; agent evaluations require us to assess an autonomous AI system that can reason, call tools, and has multiple ways of achieving its objectives.

We know agents pack power with a hint of unpredictability. With an agent, we aren’t working with a simple input/output combination; they operate over multiple turns. This makes agents useful, but it also makes them difficult to evaluate.

In traditional software development, unit and integration tests help ensure code correctness, reliability, and stability by catching bugs before they reach production. Agents introduce variability in their outputs that renders standard software testing inadequate. Holding everything constant and asking the same question twice doesn't guarantee the same result, let alone the steps required to get to the result. Using evaluations is a good mechanism to measure and limit this variability.

Another challenge is silent failures and error propagation. Agents can fail a step, continue to run steps, and return a faulty result with complete confidence. This means some errors won’t even return an error message!

Agent evaluation isn’t a one-off activity, it actually spans the entire project lifecycle. During the development phase, we will validate and test whether our agent behaves correctly by grading it on a diverse set of examples where the expected outcome is known. In production, we will monitor our agent to ensure things are running smoothly and performing well when handling real-world requests.

What do we measure?

It’s time to figure out what we want to measure. Agent performance can’t be captured in a single score so we need to measure different aspects of an agent’s performance. We define two performance areas: Operational and Behavioral. Each performance area comes with its own set of evaluation dimensions and corresponding metrics.

Operational performance

Operational performance measures whether the agent executes tasks reliably and efficiently.

Key evaluation dimensions

• Latency

• Resource usage

• Cost efficiency

• Reliability

Example metrics

• Response latency: time taken to complete actions and produce the final output.

• Memory and processing usage: CPU, GPU, and memory consumed during task execution.

• Token consumption: the number of input and output tokens used by an LLM-based system, which helps estimate system costs.

• Tool call failure rate: ratio of failed tool calls to total tool calls.

These metrics don’t require comparing the agent’s performance to a reference; they capture how the system performs.

Behavioral performance

Behavioral performance measures whether the agent behaved correctly while completing a task. We evaluate both the final response and the trajectory.

While we are building the agent, we can establish ground truth, which consists of the expected outcome and, when relevant, the expected behavior or trajectory. We then compare the agent’s actual behavior against the ground truth. The appropriate metric depends on the task. High-stakes workflows may require an exact trajectory match, while more flexible tasks may only require certain tools to appear in the correct order.

Key evaluation dimensions :

• Tool-use correctness

• Trajectory correctness

• Output correctness

Example metrics:

• Single-tool use: verifies that a mandatory tool call was used.

• Exact/strict match trajectory: requires the actual actions to match the expected actions and be in the expected order.

• Precision of trajectory: measures how many actions taken were correct according to the expected trajectory.

• Semantic match: measures whether the agent’s final response conveys the same meaning as the expected response.

How do we measure it?

Once you know what to measure, we can choose how to measure it. A robust evaluation strategy combines deterministic checks, LLM judges, and human judgment.

Deterministic evaluations should be our first choice when success can be expressed objectively. We can execute these checks using code. They can verify the metrics we defined above such as single-tool use, trajectory match, and token consumption.

They can answer questions such as:

• Did the agent produce valid JSON?

• Did it call the correct tool with the correct arguments?

• Did it stay under the token budget?

• Did the workflow complete within the latency SLA?

These evaluations are fast, cheap, and reliable. When success relies on subjective qualities, this is when the limitations of deterministic evaluations kick in.

LLM judges or LLM-as-a-judge come into play when our checks aren’t deterministic. Instead of defining rigid rules, we use one LLM to evaluate the output of another LLM/agent. LLM judges can evaluate an agent based on a rubric or trait for things like helpfulness, tone, or relevance. LLM judges provide the benefit of evaluating subjective qualities and scaling better than human evaluations, but they come with their own set of complications. They rely on an LLM, which incurs a cost each time the metric runs, and their reliability depends on the quality of the evaluation criteria. For subjective or domain-specific traits, calibration against human judgment can help ensure the LLM judge differentiates good and bad output.

These are questions LLM judges would be perfect for answering:

• Was the response helpful?

• Was the explanation complete?

• Was the tone appropriate?

• Was the reasoning sound?

• Did the agent choose a reasonable strategy?

Human judgment remains the gold standard for evaluating agent performance. Humans act as the calibration layer for the entire evaluation pipeline. It is the responsibility of a human to define what a good output looks like and why it is a good output. When starting the agent evaluation process, a Subject Matter Expert (SME) is responsible for labeling responses in an evaluation dataset and writing an explanation. This is what we will eventually calibrate LLM judges against.

Each of these strategies answers a different question. Humans define ground truth and periodically validate evaluations. Deterministic evaluations provide fast, cheap, and reliable checks. LLM judges expand our evaluation capabilities by checking subjective and complex behaviors.

Evaluating agents across the lifecycle

We know what to measure and how to measure it; now, let’s figure out when to apply evaluations. There are two distinct phases of agent evaluation: pre-deployment validation and post-deployment monitoring. The data we have available, the goals, and the techniques we use across the two phases differ. Let’s look deeper into what makes these two phases different.

Pre-deployment validation

The primary objective before deploying an agent is to ensure it meets the quality requirements for real-world use.

This is our opportunity to run validation and testing in a controlled environment. We determine what tasks an agent can do, create a set of representative test cases with expected outcomes, and define what good behavior looks like. This makes the pre-deployment phase perfect for validating both what the agent produces and how it gets to those outcomes. We can also establish performance baselines, uncover failure modes, and iterate on the agent to prevent known failures from reaching users.

At the center of validation is what is known as the evaluation dataset or golden dataset. It contains representative examples of tasks we expect the agent to handle along with the expected outcomes and behaviors required to evaluate them. Building a good evaluation dataset often requires SMEs to provide their domain knowledge to properly define correct responses and behaviors. In the case of an HR agent, an HR expert will know whether an answer actually provides the level of explanation required to handle an employee’s question or if an HR request needs to be routed through a particular system.

Once we have an evaluation dataset, we can run the agent against those test cases and measure its performance.

During pre-deployment, behavioral evaluation is particularly important because we can compare the agent’s performance to expected outcomes and behaviors. As mentioned earlier, we evaluate the final response and trajectory using a combination of deterministic checks, LLM judges, and human review. For operational evaluation, we can establish performance thresholds around latency, resource usage, and token consumption we expect the production-ready agent to meet.

Putting it all together, we create a validation loop:

1. Build a representative evaluation dataset: include common tasks, edge cases, and important workflows.

2. Define expected outcomes and behaviors: specify the agent’s desired final response and how it should behave.

3. Run the agent multiple times: since agents are nondeterministic, measure variance in responses and identify inconsistent behavior.

4. Apply multiple evaluation methods: combine deterministic checks, LLM judge traits, and human review.

5. Analyze failures: inspect final response, individual tool calls, trajectory, and other relevant metadata to diagnose failures.

6. Iterate and rerun: fix failures, update the evaluation dataset, and compare the new version against the previous baseline.

There will always be room for improvement, but that shouldn’t hold us back from deploying the agent. The objective of the pre-deployment phase is to understand how the agent behaves, where it fails, and whether the agent’s performance shows it's ready for production.

Example: Evaluating an HR agent

Let’s use an HR (Human Resources) agent to walk through the entire agent evaluation lifecycle. The goal of the HR agent is to help employees answer HR questions and automate basic HR workflows.

The agent has access to several tools that allow it to accomplish HR tasks:

• Company policy retrieval to find relevant HR policies

• PTO (Paid Time Off) balance API to retrieve an employee’s available PTO balance

• Payroll system to answer pay-related questions

• HR ticket system access to look up, submit, and route HR requests

• Employee directory lookup to identify the appropriate contact

Let’s see how the HR agent is validated before deployment and monitored after deployment.

Step 1: Build the evaluation dataset

The first step is to work with HR SMEs to create a representative set of test cases that cover edge cases and important workflows. The dataset could be as simple as this:

These examples define the answer we want to see and how the agent should get to that answer.

Step 2: Validate answers and behaviors

Next, we run the HR agent against the evaluation dataset and examine three things:

• What the agent answered with

• What the agent did

• How the system performed

For the PTO days test case, the final answer might be correct, but the actual trajectory could reveal that the agent never called the PTO API. A single-tool-use check could verify that a mandatory tool was used. For a parental-leave question that requires escalation to an HR specialist, the agent must first search the employee directory to identify the appropriate contact, then retrieve the contact’s details. This is perfect for an exact trajectory match since we require the actual trajectory to match the actions and order of the expected trajectory.

Operational metrics can reveal if the agent consistently takes too long to answer or if it consumes more tokens than we allow it to.

It’s also important to run the agent multiple times to identify any sort of inconsistent behavior across runs.

There is no single score that can represent agent performance. We must strategically choose metrics to evaluate different dimensions. This is why pre-deployment validation should combine:

Final response quality + tool use and trajectory + operational performance

Step 3: Use LLM judges and human judgment

Some HR interactions require more nuance than others. Determining if the PTO balance tool was called can be handled by a deterministic check. Determining if an explanation of a benefits policy is clear and helpful? That’s much harder to answer with deterministic checks. This is where an LLM judge can evaluate a defined trait or rubric. An LLM judge can answer questions like:

• Is the response consistent with the provided policy?

• Does the response provide enough context for the employee to understand what to do next?

• Does the response directly answer the employee’s question?

• Is the agent’s final outcome the same as the expected final outcome?

All four of these questions are individual traits. Each one has a prompt that specifies the criteria for the check. For example, the criteria for the final outcome check might direct the LLM judge to check if the agent’s answer and the reference answer are semantically equivalent with no contradictions.

SMEs should also review traits for the test cases in the evaluation dataset to validate whether the LLM judge aligns with their expert judgment. Disagreements between the SME and LLM judge help highlight weaknesses in the agent but also potential ambiguity in the evaluation criteria itself. The goal is to ultimately have your test cases pass both SME and LLM judge evaluation.

Post-deployment monitoring

Now that the agent is deployed, our evaluation focus changes.

In development, we operate in a controlled environment where we decide the test cases and define the ground truth. In production, we face unexpected requests, changing business processes, and model updates.

This is why we can’t treat production monitoring as pre-deployment validation at a larger scale. They have two completely different objectives.

The goal of the post-deployment phase is to determine whether the deployed agent remains healthy and behaves correctly when tasked with real-world requests. We need to detect operational issues, performance degradation, unexpected behaviors, and new failure modes.

A key part of monitoring is capturing the agent’s traces. A trace is a raw record of what happened during a run. It includes the user’s request, actions taken by the agent, the tools called, intermediate results, guardrails triggered, timespans, the final output, and other metadata. The final output isn’t enough; we need a detailed log of every decision and action the agent took to confirm the agent produced the correct answer without triggering guardrails, making excessive tool calls, or burning unnecessary tokens.

Operational monitoring provides the baseline. Evaluation dimensions like latency, resource consumption, token consumption, and tool-call failures allow us to establish alerts, catch cost or latency increases, and identify infrastructure problems.

Behavioral monitoring also has its role in the post-deployment phase. In production, we generally don’t have a defined reference answer or expected behavior. Instead, we extract the actual response and trajectory from the agent trace and use different evaluation methods to understand what’s going on. Deterministic checks can monitor objective characteristics. LLM judges can check for relevance, helpfulness, and other subjective traits. Even human feedback can be incorporated into the mix through thumbs up/down, escalations, support tickets, and periodic SME reviews.

Example: Evaluating an HR agent continued

Step 4: Monitor the agent in production

Once the agent meets the required validation criteria, we can deploy it into production with monitoring in place.

Monitoring our agent starts with capturing traces. We can then decompose the trace to inspect the agent’s behavior. For each interaction, we capture a trace that contains information like tool calls, final responses, technical metadata, and more. We start with tracking operational dimensions like latency, token consumption, tool failures, and resource usage to monitor the health of our system. We can also use behavioral evaluations to observe what is happening and identify interactions that require further inspection.

Step 5: Turn production failures into new test cases

Let’s say an employee tells the HR agent, “I need to change my direct deposit.” The agent then incorrectly attempts to modify the employee’s bank details instead of routing the employee to the Payroll workflow as expected.

How do we know this happened? From the agent’s trace, we know there is an unexpected tool call. An LLM judge could be used to assess improper behavior. A user could have submitted a thumbs down because the agent made a mistake with the bank details.

Once an interaction is flagged, an SME can review it to confirm that it is a genuine failure. They can confirm the root cause and we can turn the interaction into a new test case in our evaluation dataset with expected outcome and behavior documented. We could even add a guardrail that prevents the agent from performing similar actions again.

The next version of the agent can be tested against the updated evaluation dataset before the next deployment. This creates the evaluation flywheel:

The agent evaluation strategy evolves as the agent encounters more real-life interactions. Production failures become new evaluation cases which make the evaluation dataset more comprehensive and help future versions of the agent improve.

Conclusion

Agents are powerful systems that can reason, take action, and autonomously achieve goals. That same autonomy makes agents harder to trust. Evaluations give us a way to comprehensively assess the agent’s responses and behaviors.

A single score isn’t an agent evaluation. We need a comprehensive strategy to evaluate what the agent produces, how it gets there, and how reliably the system operates using a combination of methods like deterministic checks, LLM judges, and human feedback. Before deployment, we validate the agent’s performance against a representative evaluation dataset composed of test cases with known answers and behaviors. After deployment, we monitor real-world interactions, find new failure modes, and feed those back into the evaluation dataset for future validation.

Ultimately, evaluation is what separates an impressive demo from a reliable agentic system. As agents move into more critical and complex workflows, a strong evaluation strategy will be key to building systems you can confidently deploy, continuously improve, and users can trust.