Scaffold Choice Dominates Model Choice in LLM Trading Backtests
Across 540 crypto futures backtests, switching from a single-prompt agent to a ReAct-style loop produced a larger performance swing than switching models — and most configurations still lost to a passive index baseline.
We ran 540 backtests of crypto futures trading agents against 20 historical scenarios from 2025 — five regimes (bull, bear, range, volatile, recovery) across BTC, ETH, SOL, BNB, and DOGE, with three runs per combination. Five local or quantized open-weight models, two agent scaffolds: a single-prompt “simple” agent and a ReAct-style “agentic” loop with tools for market data, position state, and order placement. The full interactive results — per-scenario breakdowns, distributions, and methodology — live on the trading benchmark page.
The result that matters for agent engineering isn’t which model won. It’s how little the model identity predicted the outcome compared to scaffold choice.
The headline split
Ranked by Sharpe, the six configurations sorted almost perfectly by scaffold:
| Rank | Model | Scaffold | PnL % | Sharpe | Max DD | Hold Rate |
|---|---|---|---|---|---|---|
| 1 | gemma4-31b-local | agentic | +7.21 | 4.69 | -5.56% | 74.0% |
| 2 | gemma4-26b-local | agentic | +18.56 | 2.97 | -11.61% | 75.6% |
| 3 | qwen3.6-35b-a3b-fp8 | agentic | +15.88 | 0.18 | -25.91% | 60.4% |
| 4 | qwen3.5-35b-a3b-local | simple | -8.26 | -6.42 | -8.71% | 84.6% |
| 5 | gemma4-31b-cloud | simple | -6.27 | -8.17 | -6.13% | 86.9% |
| 6 | gemma4-26b-local | simple | -12.93 | -8.19 | -11.88% | 93.5% |
Every agentic configuration is positive. Every simple configuration is negative. The gap between the worst agentic Sharpe (0.18) and the best simple Sharpe (-6.42) is larger than the gap between any two models within the same scaffold. The scaffold variable is more discriminating than the model variable.
Same model, two outcomes
The cleanest demonstration is gemma4-26b-local, which we ran under both scaffolds. Same weights, same scenarios, same RNG seeds per run index, same market data:
- Agentic: +18.56% PnL, Sharpe 2.97
- Simple: -12.93% PnL, Sharpe -8.19
A 31-percentage-point PnL spread and a Sharpe swing of more than 11 points, driven entirely by whether the model gets a tool-use loop with intermediate observations or a single forward pass through a structured prompt. The model’s “trading ability” — whatever that means in isolation — is essentially undefined without specifying the scaffold around it. The harness is the agent, again.
The simple configurations don’t lose by making bad trades. They lose by failing to act. Hold rates of 84–94% mean the model spends most ticks deciding to do nothing, then takes large directional positions when it does move — the worst possible distribution if your decisions are noisy.
Variance is not alpha
The qwen3.6-35b agentic configuration is a useful counterexample to “agentic beats simple, end of story.” It posted +15.88% average PnL — third on the board — but a Sharpe of 0.18 and a max drawdown of -25.91%. The PnL standard deviation across runs was 36.68. One scenario (eth-bull-2025-may) returned +120.19% on average; another (sol-volatile-2025-jan) lost 17.6% with a single-run max drawdown of -64.89%.
This is what the scaffold-change looks like when the loop unlocks aggression without unlocking discipline. The agentic loop gave qwen3.6 the capacity to take leveraged directional positions on observed information. It did not give it the capacity to size those positions sensibly. The result is high mean return with risk-adjusted performance indistinguishable from a coin flip.
The Sharpe-leading gemma4-31b-local agentic took the opposite trade: lower mean PnL (+7.21%) but standard deviation of 6.59 and a max drawdown of only -5.56%. Lower ceiling, much higher floor.
A leaderboard sorted by PnL would rank qwen3.6-agentic above gemma4-31b-agentic. A leaderboard sorted by Sharpe inverts that ranking. With three runs per combo, neither ranking is statistically stable — but the shape of the variance tells you which scaffold-model pair is reliable and which is lucky.
The baseline humility check
The benchmark included three non-LLM baselines:
- Index fund (equal-weight hold across the symbol basket): +3.29% PnL, Sharpe 2.17
- TA (simple technical rule set): +0.09% PnL, Sharpe -0.70
- Random (uniform random actions): -0.01% PnL, Sharpe 0.11
Of the six LLM configurations, exactly two — both agentic gemmas — beat the index fund on risk-adjusted return. The third agentic model and all three simple models lost to passive buy-and-hold. The “best” simple agent (qwen3.5) was outperformed by random action on Sharpe and by holding nothing at all on PnL.
This is the calibration most agent benchmarks lack. Without a no-thinking baseline, the agentic-vs-simple comparison looks like a strong win for the loop. With the baseline included, the comparison becomes: scaffold choice is the difference between losing money and being competitive with a strategy that requires no model at all.
What this means for practitioners
Hold rate is a leading indicator of scaffold failure. When a model’s idle behavior is “do nothing,” a 90%+ hold rate isn’t conservatism — it’s the model declining to engage with the task. The simple agents here weren’t trading badly; they were barely trading. Any agent system where the no-op action is reachable should track hold rate (or its task-equivalent — “deferred,” “skipped,” “passed to human”) alongside outcome metrics. A high pass rate driven by high refusal rate is not a working agent.
Mean and variance need to be reported together for every scaffold-model pair. Three runs per combination is too few to publish point estimates with confidence intervals, but the standard deviations recorded here vary by more than 5x across configurations on the same scenarios. A scaffold that produces +15% mean PnL with σ=37 and a scaffold that produces +7% with σ=7 are different products, not different points on the same curve. Single-number leaderboards hide this.
Test against a passive baseline you’d be embarrassed to lose to. “Hold the basket” or “always answer yes” are not strong strategies. They’re the floor. If your agent loses to them, the scaffold isn’t helping the model — it’s hurting it. Two of three simple agents here would have been a strict improvement if replaced with while True: hold.
The model identity is less informative than people assume. A 26B model under an agentic loop posted higher PnL than a 35B model under the same loop. The same 26B model under a simple prompt was the worst configuration in the run. If you’re evaluating models in an agent context, you are evaluating the model-scaffold pair, not the model. Report both, and don’t generalize across scaffolds.
The data in this run is narrow: open-weight models in the 26B–35B range, a single asset class, 2025 backtests, three runs per combo. The specific PnL numbers don’t generalize. The structural finding — that scaffold choice moved performance more than model choice, and that most configurations still lost to a passive baseline — is what to take from it.