An agent benchmark score measures a configuration. That configuration is a model plus the tools it was allowed to call, the reasoning and output budget it was given, the code that drove it, the endpoint that served it, and the verifier that graded it. Results published between November 3 and November 14, 2026 show those surrounding settings moving scores and costs by as much as the model choice itself, and in some cases by more. A model name is therefore too underspecified to select, evaluate, or deploy against. The thing to record and check is the full configuration.
Turning on tools beat moving up a model tier
Artificial Analysis re-ran the VIALS-AA scientific image benchmark in two formats: single-shot with no tools, and agentic with code execution. Tool use nearly doubled average scores, from about 25% to about 47%. The comparison across tiers is the striking part. Claude Haiku 5.5 with tools scored higher than Claude Fable 5.2 without tools, at roughly one-tenth of the cost.
A team reading only the single-shot results would conclude it needs the larger model. A team reading only the agentic results would conclude the smaller one is enough. Both conclusions are correct for the configuration they describe. Neither transfers to a deployment that differs from it.
The accompanying write-up adds a second finding. Anthropic models increased their tool use as capability rose, up to about 37 calls per task for Fable 5.2. OpenAI models made fewer tool calls as they improved. This says nothing about which approach is better. It does mean that configuration parameters interact with model families.
Suppose a production system caps tool calls per task to bound latency or spend. That cap is a constraint the benchmark never imposed:
- A ceiling well below that per-task call count removes one family’s preferred strategy.
- The same ceiling barely affects a family that already solves tasks in fewer calls.
- The ranking you read was measured without your cap, so it may not hold under it.
The DeCAF pipeline-optimization work treats tool policy as its own dimension, separate from model selection. Its largest reported gain was a 42.45% latency reduction, alongside a 2.58-point accuracy improvement. That is further evidence that the settings around the model carry much of the leverage.
Identical open weights score differently depending on who serves them
The Endpoint Accuracy Index v1.1 tested six open-weight models across more than 100 serverless endpoints. Roughly one in four endpoints scored below the reference implementation. The causes matter more than the headline. Quantization alone did not explain the losses. They came from:
- output token caps
- reduced reasoning tokens
- tool-calling argument type errors
All three are configuration choices made by the provider. None of them is visible in the model name on a pricing page.
The third cause is especially costly for agents. A chat interface can tolerate a malformed field. In a long tool loop, the same malformed argument either triggers a retry, which adds cost, or derails the trajectory, which loses the task.
The first two causes hit hardest on the models that most depend on long generation. Thinking Machines’ Inkling 2 Small shows the pattern. It posted large agentic gains, with AutomationBench-AA rising from 5% to 46%, while using about 150K output tokens per task. A model whose agentic numbers depend on that much generation is exactly the workload where a provider’s trimmed reasoning budget or output cap does the most damage.
A useful analogy is container images. A model name served by a third party behaves like an image tag: the same tag can resolve to different contents depending on the registry. What you want is the digest. The archive’s piece on invisible model-side guardrails described closed providers silently changing what a model name resolves to. The endpoint results are the open-weight version of the same problem. The weights are identical, and the serving stack still changes what you get.
A tied score can hide a wide cost spread
In the Coding Agent Index v2.1, Claude Fable 5.2 and Claude Opus 5.5 share the lead at an index score of 58. Cost per task varies about 30x among the top configurations. Artificial Analysis frames the results in terms of configurations rather than models, and that framing is accurate.
Meta’s Muse Ember launch makes the same point from another direction. It is described as matching Claude Haiku 5.5’s best score at roughly one-third of the cost per task. The phrase “best score” admits that Haiku has more than one score, depending on how it was run.
Accuracy-only comparisons throw this information away. Two entries with equal scores can represent very different operating points:
- one cheap and fast
- one expensive and tool-hungry
- one that only reaches that score at maximum reasoning effort
This is the same arithmetic behind intra-agent model routing. That arithmetic only works if you know which configuration produced the number you are routing on.
The verifier is part of the configuration too. The index moved to Terminal-Bench 4.1 specifically for its reward-hack-resistant verifiers. A score from an earlier, more gameable verifier is a measurement of a different thing.
Proving the production configuration is the evaluated one
Anthropic’s guidance on building safety evaluations lists poor reproducibility among the common failure modes and recommends pre-registration. Applied to capability evals, pre-registration means writing down the full configuration before the run, so the result is bound to it.
Anthropic’s FlashProve attacks the other end of the gap. It is a cryptographic way to verify that a response came from a specific model. One stated use is letting evaluators confirm that the model under test matches the one in production. FlashProve attests the model. It does not, as described, attest the output cap, the reasoning budget, or the tool set. Even so, it is a concrete mechanism aimed at drift between what was evaluated and what is running.
Drift matters most where you cannot see production behavior. Legora’s CTO describes building legal agents under client-confidentiality constraints that block access to agent traces. In that setting, pre-deployment evaluation is nearly the only evidence available. Any difference between the evaluated configuration and the deployed one is invisible until it causes a failure.
Treat a configuration manifest as the key for every eval result and every production trace. At minimum, record:
- the model identifier and serving endpoint or provider
- the reasoning effort or thinking budget
- max output tokens per call
- the exposed tool set, with per-task tool-call and turn caps
- the harness version and tool-schema version
- the grader or verifier version
If any field differs between the eval manifest and the production manifest, the eval result does not carry over automatically.
The controlled-variable approach to benchmarking agent dependencies held the model fixed and swapped the search service. The same discipline applies one layer down:
- Re-check each endpoint. Before routing traffic to a third-party endpoint for an open-weight model, run a small reference set against it. Repeat when the provider announces changes.
- Validate tool arguments at the boundary. Check arguments against the schema and track type-error rate as a per-endpoint metric. The Endpoint Accuracy results suggest that rate is where serving differences show up first.
- Read the whole leaderboard row. Treat the cost and tool-call columns as part of the result, not as footnotes to it.
If your system caps tool calls or turns more tightly than the benchmark did, re-run the comparison under your caps. Model families that rely on many tool calls lose more under tight caps than the published ranking suggests.
The model name tells you which weights you meant to run. The manifest tells you what was actually measured, and whether production is still running it.
This article was generated with AI assistance and reviewed by the editors before publication.