Skip to content
agent engineering

Patterns, experiments, benchmarks

How agents are built, and what holds up when tested

Agent patterns explained in plain terms, and original experiments that test the claims, with the method published and the uncertainty stated.

Latest experiments

All experiments →
Perspectives  ·  November 14, 2026

Same Weights, Different Scores: Agent Benchmarks Measure Configurations, Not Model Names

An agent benchmark score reflects the model together with its tool access, reasoning budget, output cap, and serving endpoint, and those settings can move results and cost more than switching models, so the full configuration is what you need to pin and verify.


By Daniel Huber  ·  6 min read

An agent benchmark score measures a configuration. That configuration is a model plus the tools it was allowed to call, the reasoning and output budget it was given, the code that drove it, the endpoint that served it, and the verifier that graded it. Results published between November 3 and November 14, 2026 show those surrounding settings moving scores and costs by as much as the model choice itself, and in some cases by more. A model name is therefore too underspecified to select, evaluate, or deploy against. The thing to record and check is the full configuration.

Artificial Analysis re-ran the VIALS-AA scientific image benchmark in two formats: single-shot with no tools, and agentic with code execution. Tool use nearly doubled average scores, from about 25% to about 47%. The comparison across tiers is the striking part. Claude Haiku 5.5 with tools scored higher than Claude Fable 5.2 without tools, at roughly one-tenth of the cost.

Continue reading →

More Reading

Perspectives

Multi-Agent Splits Pay Off on Model, Clock, or Credential — Not on Role

Splitting an agent into planner, coder, and reviewer roles on the same model adds cost without adding solved tasks; a multi-agent boundary is worth building only when it separates different models, different timelines, or different permissions.

October 31, 2026
Perspectives

Benchmarking the Search API, Not the Model: Controlled-Variable Evals for Agent Dependencies

The services an agent calls are now being measured as swappable parts with model, harness, and prompt held fixed, and that experimental design is worth copying onto your own stack more than any ranking it produces.

October 17, 2026
Perspectives

The State Agents Hand Each Other Is the Unguarded Channel in Multi-Agent Systems

A demonstrated agent worm, a collusion finding from ORBIT, and two papers on stale inherited state all point at the same gap: nothing checks what flows between agents.

October 3, 2026

From the Archive

Context Layer

Context Language Models: Giving the Model Write Access to Its Own Context

An examination of Context Language Models, a University of Washington and Meta proposal that exposes an agent's live context as an editable file, and what it changes about compaction, caching, and training.

Perspectives

Decision Models: The Non-Generative Layer Slotting Under Agent Pipelines

A new class of small, structured-output-only classifiers is being wired into agent graphs for routing, scoring, and judging — decoupling 'decide' from 'generate' in ways that change both cost and evaluation coverage.

Perspectives

Three Agent Security Incidents in One Week: The Threat Model Has Field Data Now

For the first time, agent security discussions can cite specific documented incidents rather than red-team hypotheticals — and the pattern across them changes how containment should be scoped.