Need a ready-made AI chatbot for customer support? Meet Enji Chat.

AI Glossary: Key Terms

Definition of Agent evals

What are agent evals? How to evaluate AI agents in production

Agent evals are repeatable tests that measure whether an AI agent can complete a real task successfully, reliably, and within defined constraints, including where relevant the tools, actions, and intermediate steps it used to reach the result.

A useful eval includes a task, explicit success criteria, one or more runs, a grader, and a result. For agentic systems, it may also include the trace or trajectory showing how the agent reached that result.

An eval is not the same as manually trying an agent a few times. Manual checks are useful during prototyping, but they do not create a stable baseline. Repeatable evals make it possible to compare versions, detect regressions, and measure consistency over time.

What are agent evals?

Agent evals turn a real use case into a repeatable evaluation. The agent receives a task in a controlled or observable environment, attempts to complete it, and is graded against criteria defined in advance.

Agent behavior is probabilistic. The same task can produce different tool choices, intermediate steps, or outputs across runs, so one successful attempt says little about reliability.

Sometimes the final outcome is enough. In other cases, the execution path matters too, for example whether the agent used the correct tool or respected an approval requirement.

Anthropic describes agent evaluations in terms of tasks, trials, graders, transcripts, and outcomes, with repeated trials useful when model behavior varies across runs.

How are agent evals different from LLM evals?

A simple LLM eval often looks like this:

Input → Response → Grade

An agent can involve a much longer chain:

Goal → Context → Tool calls → Actions → State changes → Outcome

An agent may select the wrong tool, pass incorrect parameters, enter a loop, skip a required step, perform an unsafe intermediate action, or reach the correct answer through an inefficient path.

Evaluating only the final text can therefore miss important failures. The answer may sound correct even when the system changed the wrong record.

Agent evaluation often needs to test the whole task, not only the final response. That can include the end state, tool usage, trajectory, constraints, and consistency across repeated runs.

What should agent evals measure?

There is no universal metric set. Evaluation depends on the task and the failures that matter.

Task success

The first question is whether the agent completed the actual task. If the goal was to create a Jira issue, update a record, or prepare a release report, the eval should verify that result rather than infer success from a confident response.

Output quality

When the task produces a report, recommendation, or other text, useful criteria can include correctness, completeness, groundedness, and relevance.

Tool use

For tool-using agents, evaluation can check tool selection, arguments, call success, and correct use of returned information. Microsoft separates these process checks from final system-level evaluation.

Trajectory

The trace can reveal unnecessary steps, repeated tool calls, loops, incorrect ordering, or poor recovery after a failed action.

Reliability

Repeated or similar cases show whether the agent succeeds consistently rather than occasionally.

Constraints and safety

An eval can also verify that the agent respected permissions, followed required approval points, and avoided prohibited actions.

Efficiency

Where it materially matters, teams can also track latency, retries, excessive tool calls, or cost.

Outcome evals vs trajectory evals

An outcome eval asks:

Did the task finish in the correct state?

For example, it can verify that the correct Jira issue was created, a report contains the required information, or a workflow reached the expected final state.

A trajectory eval asks:

Did the agent follow an acceptable path?

It may check whether the agent used appropriate tools, avoided unnecessary repeated calls, respected an approval point, or stayed within the required resource scope. Not every task needs one exact sequence. Different valid paths can reach the same correct result, so overly strict trajectory grading can reject useful behavior.

A practical eval focuses on required actions, critical constraints, prohibited behavior, and the outcome, while allowing variation where multiple paths are acceptable.

How are agent evals graded?

Different parts of a task need different graders.

Deterministic checks work well when success can be verified exactly. They can test an expected value, confirm that a tool was called, inspect database state, or check whether a software test passed.

Rubric-based evaluation is useful when quality depends on several defined criteria rather than one exact answer.

LLM-as-a-judge can evaluate open-ended qualities such as completeness, relevance, or groundedness when exact matching is impractical. However, model-based judges are themselves probabilistic. They need clear rubrics and validation against human judgment rather than being treated as automatically correct.

Human evaluation remains useful for ambiguous or high-stakes cases and for calibrating automatic graders.

One eval can combine several graders. The grader should match the part of the task being assessed. Anthropic likewise distinguishes code-based, model-based, and human graders, with different strengths and limitations.

Offline vs online agent evals

Offline evals run against predefined tasks before deployment or before a new version is released. They are useful for regression testing, comparing models or prompts, checking tool changes, and deciding whether a release still meets the expected quality bar.

Online evals assess production traces. They help identify failure modes that were missing from the original dataset and show how the agent behaves on real inputs.

The two should reinforce each other. A production failure can become a new eval case, the team can fix the problem, and the case can then remain in the offline suite as a regression test. This turns real-world failures into reusable evidence for future releases.

The useful cycle is:

Production failure → new eval case → fix → regression test

LangSmith, for example, supports evaluation both on offline datasets and on production traces.

How do you build a useful agent eval set?

An eval set can start small. It does not need to be a large benchmark before deployment.

  1. Choose a real task the agent is expected to perform.
  2. Define what successful completion means.
  3. Add representative normal cases.
  4. Add important edge cases and known failure modes.
  5. Decide which dimensions matter: outcome, tools, constraints, and, where relevant, efficiency.
  6. Choose graders that match those dimensions.
  7. Run multiple trials when behavior varies between runs.
  8. Add real production failures as they appear.

A small representative set tied to actual use is usually more informative than a large synthetic collection with weak connection to production.

The set should evolve as new tools, models, workflows, and production behavior reveal new failure modes.

Example: evaluating a project-management agent

Suppose an agent receives this task: Identify what is putting the release at risk and prepare a stakeholder update.

It can use Jira, Git, Slack, and meeting notes. A weak eval would ask only whether the report sounds plausible.

A better eval would verify the correct blocker, current sources, the right milestone, no unsupported claims, required report sections, and project permissions. Repeated runs should remain acceptably reliable. If the agent can take actions, the eval should check the resulting state too. Did it update the correct record? Did it request approval when the action required one?

This is the central distinction: response quality is not the same as full agent performance.

The same logic also applies when evaluating an AI tool before buying it. Test it against real tasks, explicit success criteria, relevant failure cases, and evidence rather than relying only on feature lists or scripted demos.

Image.

Agent evals vs LLM evals: LLM evals often focus on a response or model behavior. Agent evals can evaluate the full task execution, including tools, actions, state changes, and outcome.

Agent evals vs Benchmarks: A benchmark is a standardized shared test. An agent eval is typically tied to a specific use case and its success criteria.

Agent evals vs Testing: Testing is the broader software concept. Evals are especially useful for probabilistic behavior that cannot always be checked with a simple binary assertion. Production agents often need both conventional tests and evals.

Agent evals vs Observability: Observability asks what happened. Evals ask whether what happened was good enough.

Key Takeaways

  • Agent evals are repeatable tests of real task performance, not a few manual trials.
  • Final-answer quality is only one part of agent performance; outcomes, tool use, trajectory, constraints, and reliability may also matter.
  • Outcome evals check whether the task finished correctly, while trajectory evals check whether the path was acceptable.
  • Useful evals combine deterministic, model-based, and human grading where appropriate.
  • Offline and production evaluation work best as a loop in which real failures become future regression cases.

Created by

Diana Ruslanova, Content Creator.

Diana Ruslanova

Copywriter

FAQ

What are agent evals?

How are agent evals different from LLM evals?

What should an agent eval measure?

What is a trajectory eval?

What is LLM-as-a-judge?

How many eval cases do you need?

Should agent evals run before or after deployment?

How are agent evals different from observability?

Can evals measure tool use?

How do you evaluate a non-deterministic agent?

Last updated in October 2026