It failed in ways I didn’t expect.
Instead of model quality issues, most failures came from system-level problems. A few examples from a small test suite:
- Broken URLs in tool calls → score dropped to 22
- Agent calling localhost in a cloud environment → got stuck at 46
- Real CVEs flagged as hallucinations → evaluation issue, not model issue
- Reddit blocking requests → external dependency failure
- Missing API key in production → silent failure
Each run surfaced a real bug, but not the kind I was originally trying to measure.
What surprised me is that evaluating agents isn’t just about scoring outputs. It’s about validating the entire system: tools, environment, data access, and how the agent interacts with all of it.
In other words, most of the failure modes looked more like software bugs than LLM mistakes.
This made me think that evaluation loops for agents should look more like software testing than benchmarking: - repeatable test suites - clear pass/fail criteria - regression detection - root cause analysis
Otherwise it’s very easy to misattribute failures to the model when they’re actually coming from somewhere else.
I ended up building a small tool to structure this process, but the bigger takeaway for me is how messy real-world agent evaluation actually is compared to standard benchmarks.
Curious how others are approaching this, especially in production settings.