The next phase in AI agent development may be shaped less by what we add than by what we learn to remove. We expect agent development to shift from building systems that can complete a task to systematically proving how well they do it and at what cost.
Organizations will recognize that a bigger model, more tools or added memory won’t automatically mean a better agent. The advantage comes from systematic evaluation in measuring quality, cost and latency, testing assumptions and keeping only the changes that deliver measurable value.
What does that shift mean for how you build a better agent? It starts with recognizing that getting an agent to work is not the same as knowing how well it performs. Building an agent often follows a familiar pattern. You start with an idea of what the agent should do, which documents it should access and which tools it should use. After you connect everything and make some refinements, the agent successfully completes its task. The job is done, right?
In our experience, this is only the beginning. Perhaps the agent works in simple scenarios but hallucinates in difficult, unseen ones. Maybe it produces inconsistent results, costs too much to operate or takes too long to respond. While other questions remain, like which model should you use? Which prompts are most effective? Which tools are actually necessary?
Just as traditional software teams rely on automated tests and performance metrics to understand how their software performs, agent developers need an equivalent discipline: systematic evaluation.
Systematic evaluation is the way forward
We put our theory into practice. In this blog, we describe how we used an evaluation-driven development process in SAS Retrieval Agent Manager (RAM) to optimize an industrial maintenance agent. Through our experiments, we improved its quality score by 68%, reduced cost per task by 95% and cut response latency by 61%.
To evaluate the maintenance agent, we needed to measure both the quality of its answers and the efficiency with which it produced them. Our framework addressed this with two complementary metric types. The first is deterministic metrics, such as cost, latency, token usage and tool usage. These measurements are fast, inexpensive and easy to compare across experiments. The second is LLM-as-a-judge metrics, in which an independent model evaluates the agent’s outputs and execution traces, including tool calls, retrieved documents and prompts. These metrics allow us to ask questions like: did the agent identify a plausible root cause, and did it support its conclusions with evidence?
We applied these metrics through an evaluation-driven development process. The process involves establishing a baseline, identifying weaknesses through evaluation results and then tracing them, implementing changes, and re-evaluating their impact.
Establishing a baseline
To establish a baseline, we used 28 real-world maintenance scenarios and measured quality, cost and latency. Quality was calculated as the average of three LLM-as-a-judge metrics:
- Task completion. Did the agent identify a root cause and produce an actionable maintenance plan?
- Faithfulness. Did it use the evidence it gathered correctly?
- Hallucination. Did it make claims unsupported by the evidence?
The baseline agent achieved a quality score of 0.562/1.0, cost $0.84 per task and took 78 seconds per task. These results provided a benchmark for evaluating future changes to the agent.
Iterating with evaluation
We created six iterations of the industrial maintenance agent, each informed by observations from evaluation.
Across the seven versions, V5 provided the best overall balance of quality, cost and latency. Quality improved from 0.562 to 0.942, while cost per task fell from $0.84 to $0.046 and latency decreased from 78 to 31 seconds. The following table shows the full results by version, along with the rationale for each change.
| Version | Quality | Cost per task | Duration | Change | Rationale |
|---|---|---|---|---|---|
| v0 | 0.562 | $0.84 | 78.01s | Baseline | – |
| v1 | 0.833 | $0.71 | 56.98s | Return structured JSON instead of HTML | Agent should focus on reasoning, not presentation |
| v2 | 0.943 | $0.38 | 42.76s | Add purpose-built database tools | Eliminate invalid SQL query retries and schema discovery tool calls |
| v3 | 0.933 | $0.41 | 37.15s | Move sensor threshold logic to code | Deterministic rules should not consume LLM tokens |
| v4 | 0.921 | $0.53 | 62.42s | Split high-level tasks into sub-agents | Specialized agents may improve quality |
| v5 | 0.942 | $0.046 | 30.66s | Use task-appropriate models | Match model capability to task complexity |
| v6 | 0.924 | $0.08 | 35.72s | Add long-term memory | Prior experience may improve future decisions |
Design patterns worth keeping
Across these experiments, three design patterns emerged as particularly effective. Not every change improved performance, reinforcing why evaluation is necessary for agent development.
1. Eliminate unnecessary LLM work
First, we stopped asking the agent to generate a static HTML report. The user-facing application already handled the presentation, so we updated the agent to return structured JSON with the root cause analysis and maintenance plan. This simplified the system prompt, allowing the model to focus on reasoning and resulting in a 48% increase in quality.
We also moved deterministic sensor-threshold classification from the system prompt into Python code. Rather than asking the LLM to apply rules such as “current above 0.55 A is critical,” we supplied the classifications directly. Quality was similar to the preceding version, but task completion was slightly higher and the behavior was more consistent.
2. Build purpose-built tools
The agent’s execution traces showed that generic SQL access led to repeated schema discovery, invalid queries, retries and unnecessary tool calls. We replaced it with custom MCP database tools that exposed the specific business operations the agent needed while handling SQL internally. This produced another major improvement in quality, cost and latency.
3. Decompose the workflow and use appropriate models
Not all architecture changes led to immediate improvement. When we re-architected the solution to divide its responsibilities among multiple focused agents, it performed worse than our previous single-agent experiments.
Although the multi-agent design initially reduced performance, it created stages that could be optimized independently. We retained an advanced reasoning model for root-cause analysis and used a smaller model for maintenance planning. This produced the best overall implementation, reducing cost from $0.376 to $0.046 per task while maintaining nearly identical quality.
4. Test assumptions
For the final experiment, we added long-term memory capabilities so that, in theory, the agent could learn from its experiences and accomplish its task more efficiently over time.
We found that these capabilities did not improve performance. Although the overall quality score was nearly the same as in previous versions, cost and latency were slightly higher due to overhead from the memory context and tools. Memory added complexity without delivering measurable benefits for this use case, so we excluded it from the final design.
From individual experiments to a discipline for better agents
Our experiments showed that optimization is as much about deciding what to leave out as what to build. Purpose-built tools and task-appropriate models earned their place. Long-term memory did not deliver measurable benefits for this use case. Even the multi-agent design required further refinement before it produced the best overall implementation. Evaluation made those trade-offs visible and gave us a basis for choosing among them.
These results reflect one maintenance use case, not a universal blueprint. They nevertheless illustrate why we expect systematic evaluation to become a foundation of agent development rather than a final check after the build. For teams, the practical starting point is to define success, establish a baseline, inspect traces, change one meaningful variable at a time and measure the result. No single model or architecture will be best for every task. The discipline is knowing which choices improve your agent. And having the evidence to explain why.
