In conversations with technology leaders, I keep encountering the same gap: an AI system is entering use, but its team has no reliable way to assess the quality of its work. My concern is what this can do to confidence in the technology. If users encounter errors and we cannot explain their causes, disappointment can become distrust.

Evaluation gives us a way to investigate those failures and decide what responsibility the system can reasonably carry. It means testing representative work against criteria agreed with the people who understand the business process.

A consistent answer can still use the wrong definition

Consider an assistant that answers questions about commercial margin. A plausible number is insufficient. The result depends on the treatment of returns, discounts, reporting periods, and the transactions included in the calculation.

The business must define those terms before the team can judge the answer. Repeating the same calculation reliably would not make an incorrect definition acceptable. If the question leaves a material ambiguity unresolved, asking for clarification may be the right response.

This is a useful starting point for evaluation: identify what an acceptable answer means, what evidence supports it, and which mistakes would change a business decision.

Evaluation must cover the components and the complete service

An AI agent combines a model with instructions, access to information, and software tools. A failure can occur in any of those parts or in the way they work together. I would examine three levels:

  • Components: does each part perform its task? For a document assistant, does the search retrieve the relevant, current material?
  • Interactions: do information and constraints survive between steps? An uncertainty identified by one agent must remain visible to the next.
  • Business outcome: does the complete service produce a correct, authorized, timely result that someone can use without excessive correction?

Component scores help locate weaknesses. They cannot simply be added together to establish the reliability of the assembled service. The complete workflow needs its own tests.

The distinction also applies to an agent’s report of success. Anthropic’s guidance on agent evaluation recommends checking the actual outcome separately from what the agent says happened. An agent claiming that it completed an action is insufficient evidence that the action succeeded.

Test results should change what the agent is allowed to do

Start with a limited scope that can be evaluated credibly. Business experts should provide representative tasks, expected results, material errors, and situations requiring clarification or escalation. Include difficult cases and requests the system should decline.

Then connect the results to an operational decision. Reliable performance on a defined task may justify delegation within that scope. Other tasks may still need human approval or remain outside the service. Permissions and spending limits must be enforced by the surrounding software; an instruction in a prompt is insufficient.

Repeat the relevant evaluations after changes to models, instructions, tools, or data. Sample real usage as well, because a test set cannot anticipate every situation. Human review belongs in the assessment: a service that needs constant correction may offer little benefit even when its final answers are accurate.

Retaining evaluation assets makes replacement possible to assess

Technology independence also depends on the ability to judge an alternative. Keep business definitions, representative test cases, and acceptance criteria under the organization’s control. A replacement can then be assessed against the same requirements, including quality, supervision, cost, and response time.

For the next deployment decision, ask for evidence against four questions:

  1. What work has the system demonstrated it can perform acceptably?
  2. Where must it ask for help or stop?
  3. Who can restrict its permissions when performance deteriorates?
  4. Can an alternative be tested against the same requirements?

These questions give evaluation a practical purpose: deciding what the organization is prepared to delegate and retaining the means to revise that decision.

I explore the methods behind this approach in “If you can’t evaluate an AI agent, why did you deploy it?” on Synthetic Horizons. The full article examines test design, supervision costs, and the relationship between evaluation and technological sovereignty in greater depth.