Skip to content
agent engineering

Perspectives

Benchmarking the Search API, Not the Model: Controlled-Variable Evals for Agent Dependencies

The services an agent calls are now being measured as swappable parts with model, harness, and prompt held fixed, and that experimental design is worth copying onto your own stack more than any ranking it produces.

By Daniel Huber · AI-generated, human-reviewed October 17, 2026  ·  7 min read

Agent benchmarks have long ranked models. In the two weeks ending October 17, 2026, several of the most useful new benchmarks held the model fixed and ranked something else: the search API, the browser driver, the inference hardware. That is a different instrument. It treats the services an agent calls as dependencies whose quality can be isolated, priced per task, and swapped under test.

The leaderboards these benchmarks produce will age quickly. The experimental design behind them is the durable part, and practitioners should copy it.

Swapping one dependency while everything else stays put

The clearest example is the Artificial Analysis Search Index. It runs 26 search API products from 13 providers through an agentic search setting with the model, harness, and prompt held constant. The only thing that changes is the search provider. Answer quality is scored on DeepSearchQA, BrowseComp, and AA-Omniscience, and cost and latency are reported per task.

The WebdriverIO Browser Agent Benchmark applies the same design to browser tooling. It uses 100 real tasks from Online-Mind2Web, keeps the model, harness, and prompt constant, and swaps only the browser tool. An independent o4-mini judge grades every run.

Nous Research’s Hermes Index inverts the setup. It fixes the harness and varies the model. The pattern is the same, though: one free variable, everything else pinned.

This is how a team already compares models before switching providers. What is new is applying the same discipline to the layer underneath the model. The tool use literature has focused on how agents call tools. These benchmarks ask a different question: given a fixed calling pattern, how much does the choice of tool vendor move the outcome? If you have read Dynamic Filtering for Web Search Agents, you already know that search result quality shapes everything downstream in an agentic RAG loop. These benchmarks put a measurement on that.

BenchmarkHeld constantVariedReported per taskGrading
AA Search IndexModel, harness, promptSearch API (26 products)Quality, cost, latencyBenchmark answer sets
WebdriverIO Browser Agent BenchmarkModel, harness, promptBrowser toolTask successo4-mini judge, judgments published
Hermes IndexHarness, four suitesModelScore, costSuite verifiers
AA-AgentPerfReplayed coding-agent trajectoriesHardwareThroughput per megawattSystems measurement

Per-query price is the wrong number to compare

Search and browser vendors price per call. Agents consume them per task, and those two numbers can diverge.

Consider what happens when a search provider returns weaker results:

  • The agent issues more follow-up queries.
  • It pulls more snippets into context.
  • It spends more model tokens reasoning over noise before it can answer.

A provider that is cheaper per query can therefore be more expensive per completed task. That holds even before counting the tasks that fail outright. Reporting cost and latency per task, as the Search Index does, captures that whole loop rather than the line item on the vendor’s invoice.

The Hermes numbers show how wide per-task cost spreads can be even among frontier models running in one harness:

ModelScoreCost per task
Claude Opus 5.563.31$4.99
GPT-6 Astra56.25$11.61
Claude Sonnet 5.553.14$2.82

In that table the highest-scoring model is not the most expensive one. Per-task measurement surfaces that kind of result, and list prices cannot predict it. The same accounting belongs on every external dependency an agent loops over.

Tip
When you evaluate a search, browser, or retrieval vendor, measure total cost per successful task: vendor fees plus the model tokens the agent spends because of that vendor’s output. Do not compare per-call prices.

A ranking is only valid for the stack it was measured on

Holding the model and harness constant is what makes these benchmarks clean. It is also what limits how far their results transfer.

A search provider that ranks first under one model’s query style may not rank first under yours. Models differ in how many queries they issue, how they phrase them, and how well they tolerate noisy snippets. Harnesses differ in how they truncate and filter results before the model sees them.

The archive has already documented how much the surrounding structure can move outcomes. Scaffold Choice Dominates Model Choice in LLM Trading Backtests found scaffold effects large enough to reorder model rankings. There is little reason to expect tool rankings to be immune to the same interaction effects.

The practical reading is to use the published leaderboards as a shortlist and the published methodology as a template. WebdriverIO makes the second part easy: all prompts, data, and judgments are published, so the setup can be rerun against your own model and harness.

The natural source of tasks is your own production traces. Sample a fixed set of real tasks, replay them with one dependency swapped, and compare. This is the same loop described in Production Traces Are Becoming the Training Signal for Agent Fixes, aimed at vendor selection instead of bug fixing. Rerun it when a vendor ships a ranking change, because a silent upgrade to a search backend is a dependency change even if no line of your code moved.

The workload model is part of the result

AA-AgentPerf extends controlled comparison to hardware. Its headline result is that NVIDIA’s Vera Rubin NVL72 delivers up to 5.7x throughput per megawatt over GB300. The more instructive detail is how the benchmark builds its workload: it replays real coding-agent trajectories, including long cached contexts and tool-call delays.

This is not a neutral design choice. Agent inference is dominated by repeated long prefixes and punctuated by idle time while tools run. A benchmark built from independent prompt-response pairs would measure a workload that agents do not produce.

The same property shows up on the client side. Claude Code now compacts long idle sessions just before the prompt cache expires, at roughly one hour, with a default threshold of 200k tokens. That feature exists because idle gaps change what a session costs when it resumes. Timing is part of the workload, not noise around it.

When you replay traces to compare dependencies, preserve the timing as well as the content. A comparison that strips out idle gaps will mismeasure anything sensitive to prompt caching. That includes the model provider, the inference host, and any tool slow enough that the cache goes cold while the agent waits.

The judge sets the resolution of the experiment

Swapping one dependency often produces small differences. When differences are small, the grader’s noise and bias can be as large as the effect you are trying to measure.

The new benchmarks handle grading in different ways:

Your internal dependency evals need the same humility. Viv Trivedy’s method for evaluating evals is a workable starting point:

  1. Run agents of varying capability on the eval.
  2. Have a review agent inspect trajectories with access to the environment and verifier.
  3. Give the review agent a skill listing known pitfalls, such as an overly strict judge.
  4. Finish with human review.

If a weak agent and a strong agent score the same, the eval cannot tell you whether search provider A beats provider B. The same applies when the judge rejects correct answers on formatting grounds.

Cheaper judges make it affordable to run enough trials to separate real differences from noise. LangChain has added TypeSafe AI’s Jev to LangSmith evals. It returns typed probabilities instead of free text, as a faster and cheaper alternative to LLM-as-a-judge.

The Artificial Analysis Decision Index shows where such models fit. They match language models on classification and tool-calling at much lower cost and latency, but trail substantially on knowledge and reasoning. That points to a usable split:

  • A typed decision model can grade a judgment that is genuinely classification-shaped, such as “did the agent reach the right page.”
  • A judgment that requires reasoning about whether an answer is correct still needs a stronger grader. Otherwise the cheaper judge is setting the floor on what your experiment can detect. Self-Attribution Bias is a reminder that grader choice carries its own systematic errors.

The benchmarks published in early October 2026 are best read as worked examples of an evaluation method: pin everything, free one variable, price the whole task, replay realistic timing, and audit the grader. The specific rankings describe someone else’s stack. The method transfers to yours.

Tags: perspectivesevaluationbenchmarkstool-usesearchcost

This article was generated with AI assistance and reviewed by the editors before publication.