Agent benchmarks have long ranked models. In the two weeks ending October 17, 2026, several of the most useful new benchmarks held the model fixed and ranked something else: the search API, the browser driver, the inference hardware. That is a different instrument. It treats the services an agent calls as dependencies whose quality can be isolated, priced per task, and swapped under test.
The leaderboards these benchmarks produce will age quickly. The experimental design behind them is the durable part, and practitioners should copy it.
Swapping one dependency while everything else stays put
The clearest example is the Artificial Analysis Search Index. It runs 26 search API products from 13 providers through an agentic search setting with the model, harness, and prompt held constant. The only thing that changes is the search provider. Answer quality is scored on DeepSearchQA, BrowseComp, and AA-Omniscience, and cost and latency are reported per task.
The WebdriverIO Browser Agent Benchmark applies the same design to browser tooling. It uses 100 real tasks from Online-Mind2Web, keeps the model, harness, and prompt constant, and swaps only the browser tool. An independent o4-mini judge grades every run.
Nous Research’s Hermes Index inverts the setup. It fixes the harness and varies the model. The pattern is the same, though: one free variable, everything else pinned.
This is how a team already compares models before switching providers. What is new is applying the same discipline to the layer underneath the model. The tool use literature has focused on how agents call tools. These benchmarks ask a different question: given a fixed calling pattern, how much does the choice of tool vendor move the outcome? If you have read Dynamic Filtering for Web Search Agents, you already know that search result quality shapes everything downstream in an agentic RAG loop. These benchmarks put a measurement on that.
| Benchmark | Held constant | Varied | Reported per task | Grading |
|---|---|---|---|---|
| AA Search Index | Model, harness, prompt | Search API (26 products) | Quality, cost, latency | Benchmark answer sets |
| WebdriverIO Browser Agent Benchmark | Model, harness, prompt | Browser tool | Task success | o4-mini judge, judgments published |
| Hermes Index | Harness, four suites | Model | Score, cost | Suite verifiers |
| AA-AgentPerf | Replayed coding-agent trajectories | Hardware | Throughput per megawatt | Systems measurement |
Per-query price is the wrong number to compare
Search and browser vendors price per call. Agents consume them per task, and those two numbers can diverge.
Consider what happens when a search provider returns weaker results:
- The agent issues more follow-up queries.
- It pulls more snippets into context.
- It spends more model tokens reasoning over noise before it can answer.
A provider that is cheaper per query can therefore be more expensive per completed task. That holds even before counting the tasks that fail outright. Reporting cost and latency per task, as the Search Index does, captures that whole loop rather than the line item on the vendor’s invoice.
The Hermes numbers show how wide per-task cost spreads can be even among frontier models running in one harness:
| Model | Score | Cost per task |
|---|---|---|
| Claude Opus 5.5 | 63.31 | $4.99 |
| GPT-6 Astra | 56.25 | $11.61 |
| Claude Sonnet 5.5 | 53.14 | $2.82 |
In that table the highest-scoring model is not the most expensive one. Per-task measurement surfaces that kind of result, and list prices cannot predict it. The same accounting belongs on every external dependency an agent loops over.
A ranking is only valid for the stack it was measured on
Holding the model and harness constant is what makes these benchmarks clean. It is also what limits how far their results transfer.
A search provider that ranks first under one model’s query style may not rank first under yours. Models differ in how many queries they issue, how they phrase them, and how well they tolerate noisy snippets. Harnesses differ in how they truncate and filter results before the model sees them.
The archive has already documented how much the surrounding structure can move outcomes. Scaffold Choice Dominates Model Choice in LLM Trading Backtests found scaffold effects large enough to reorder model rankings. There is little reason to expect tool rankings to be immune to the same interaction effects.
The practical reading is to use the published leaderboards as a shortlist and the published methodology as a template. WebdriverIO makes the second part easy: all prompts, data, and judgments are published, so the setup can be rerun against your own model and harness.
The natural source of tasks is your own production traces. Sample a fixed set of real tasks, replay them with one dependency swapped, and compare. This is the same loop described in Production Traces Are Becoming the Training Signal for Agent Fixes, aimed at vendor selection instead of bug fixing. Rerun it when a vendor ships a ranking change, because a silent upgrade to a search backend is a dependency change even if no line of your code moved.
The workload model is part of the result
AA-AgentPerf extends controlled comparison to hardware. Its headline result is that NVIDIA’s Vera Rubin NVL72 delivers up to 5.7x throughput per megawatt over GB300. The more instructive detail is how the benchmark builds its workload: it replays real coding-agent trajectories, including long cached contexts and tool-call delays.
This is not a neutral design choice. Agent inference is dominated by repeated long prefixes and punctuated by idle time while tools run. A benchmark built from independent prompt-response pairs would measure a workload that agents do not produce.
The same property shows up on the client side. Claude Code now compacts long idle sessions just before the prompt cache expires, at roughly one hour, with a default threshold of 200k tokens. That feature exists because idle gaps change what a session costs when it resumes. Timing is part of the workload, not noise around it.
When you replay traces to compare dependencies, preserve the timing as well as the content. A comparison that strips out idle gaps will mismeasure anything sensitive to prompt caching. That includes the model provider, the inference host, and any tool slow enough that the cache goes cold while the agent waits.
The judge sets the resolution of the experiment
Swapping one dependency often produces small differences. When differences are small, the grader’s noise and bias can be as large as the effect you are trying to measure.
The new benchmarks handle grading in different ways:
- WebdriverIO uses an independent o4-mini judge and publishes every judgment. Its grading can be audited rather than trusted.
- AA-3D ranks models by Elo from blind pairwise human votes across 55 tasks. It reports Claude Sonnet 5.5 at 1194, statistically tied with Claude Opus 5.5, GPT-6 Astra, and Claude Fable 5.1. That is an honest statement that the instrument cannot separate those four entries.
Your internal dependency evals need the same humility. Viv Trivedy’s method for evaluating evals is a workable starting point:
- Run agents of varying capability on the eval.
- Have a review agent inspect trajectories with access to the environment and verifier.
- Give the review agent a skill listing known pitfalls, such as an overly strict judge.
- Finish with human review.
If a weak agent and a strong agent score the same, the eval cannot tell you whether search provider A beats provider B. The same applies when the judge rejects correct answers on formatting grounds.
Cheaper judges make it affordable to run enough trials to separate real differences from noise. LangChain has added TypeSafe AI’s Jev to LangSmith evals. It returns typed probabilities instead of free text, as a faster and cheaper alternative to LLM-as-a-judge.
The Artificial Analysis Decision Index shows where such models fit. They match language models on classification and tool-calling at much lower cost and latency, but trail substantially on knowledge and reasoning. That points to a usable split:
- A typed decision model can grade a judgment that is genuinely classification-shaped, such as “did the agent reach the right page.”
- A judgment that requires reasoning about whether an answer is correct still needs a stronger grader. Otherwise the cheaper judge is setting the floor on what your experiment can detect. Self-Attribution Bias is a reminder that grader choice carries its own systematic errors.
The benchmarks published in early October 2026 are best read as worked examples of an evaluation method: pin everything, free one variable, price the whole task, replay realistic timing, and audit the grader. The specific rankings describe someone else’s stack. The method transfers to yours.
This article was generated with AI assistance and reviewed by the editors before publication.