By Daniel Huber · original experiment · October 10, 2026 · 9 min read
The short version
The agents traded five cryptocurrencies on real prices with paper money (simulated, no real funds at stake). After each closed trade, a second system looked at what had gone wrong, wrote the agents new rules, and kept the rules that would have done better on past prices.
Simply buying the five coins and holding them beat every version of it. To find out why, we hid patterns in made-up price data and checked which ones the loop would spot: only those already on its list of 38 conditions to check. Every other kind of pattern went unnoticed, in all 90 attempts.
What this means for anyone building agents: a loop that reflects on its mistakes is not automatically a loop that learns something new. Before trusting one, plant a pattern it was never told about and see whether it finds it.
Verdict
Over ten months of paper trading, no version of the loop beat a passive index fund. The last test shows why it could not learn its way there: the loop sees the market only through a scan of 38 fixed conditions, and patterns planted outside that list were found in 0 of 90 tapes.
Original measurement0 of 90
planted patterns outside the library found
0 of 18
pair-trading settings that passed
+4.59%
best phase 2 arm in 5.4 days; the index fund made +37.08%
Discovery: 150 synthetic tapes, 10 seeds per cell · Trading: paper money on live prices, every Sharpe interval spans zero · One market, one loop design
The loop
In short: the agents traded, and a set of programs around them tried to turn their mistakes into better instructions.
The agents traded perpetual futures on the Kraken exchange (contracts that follow a coin’s price and never expire) for bitcoin, ether, solana, dogecoin and XRP. Each started with $10,000 of paper money and made one decision every 15 minutes; trading fees, funding payments and forced liquidations were simulated as on the real exchange. Each agent was judged by its risk-adjusted edge over an index fund that simply holds the five coins: how much more it earned, allowing for how bumpy its returns were.
Around the agents ran the loop. A trade journal recorded every decision. After each closed trade, a language model wrote a reflection on what had happened; recurring failures were grouped into clusters; from those clusters the loop wrote skills, short instructions added to the agent’s prompt, and generated changes to the agent’s code. Every change was replayed against recorded prices first and kept only if it would have helped. By 20 September 2026 the arena had logged over 73,000 decisions, 4,189 trades and 1,736 reflections.
The experiments
Six tests over ten months; none showed an advantage that held up.
How to read the table: the Sharpe ratio measures return per unit of risk, and its interval is the range the true value plausibly lies in. An interval that includes zero means the result cannot be told apart from luck. A test is pre-registered (its pass rule written down before the data existed) from PE-1 on; FAIL means that rule was not met.
| When (2026) | What was tested, and the result | Verdict |
|---|---|---|
| May – 13 Sep | Phase 1, five Gemma 4 agents with and without parts of the loop, live. Over the last 30 days every agent, and a rule-based bot, trailed the index fund by 8 to 27 points; every Sharpe interval spanned zero. In a 62-day comparison, the agent with skills made +5.85% and its twin without them −4.09%, while the index fund made +26.91%. | FAIL |
| 18 – 22 Aug | PE-1, a synthetic market with one planted pattern: a 4-hour drop of 2% or more is followed by a rise. Over 30 learning cycles, none of 30 skills described the pattern and none of 35 planted opportunities was traded. | FAIL |
| 13 – 19 Sep | Phase 2, GLM-5.3 Flash with and without skills, live. After 5.4 days: +4.59% against −2.09%, while the index fund made +37.08%. Stopped by decision, far short of its 60-day window. | No verdict |
| 13 – 18 Sep | PE-2, the planted test again, with the market scan added. A skill named the pattern on day 1, but the agent traded 5 of 11 planted events in the scoring window (50% needed) and lost money trading the planted symbol (−$2.99 per closed trade). | Knowledge only |
| 20 Sep | Q1, a pair-trading backtest (betting on the gap between two related coins, tested on past prices): 30 perpetuals, 67 pairs, 18 settings, a 144-day holdout. No setting passed. The default setting returned −30.70% (Sharpe −0.68, interval −3.80 to 2.45) while the index fund made +26.59%. | FAIL |
| 20 Sep | Discovery test: does the market scan find patterns that are not in its library? | LOOKUP |
Phase 1 had already retired the profitability question on 13 September. Fifteen-minute price data on crypto futures is among the most heavily mined information in finance, so a loop that cannot find a pattern planted on purpose has little chance there. The remaining tests asked the narrower question: can the loop learn anything it was not given?
The discovery test
This test checked whether the loop can notice a pattern nobody told it to look for.
PE-1 showed that every stage of the loop summarised the agents’ own behaviour, never the market. So a market scan was added: it tests 38 fixed conditions, such as “the price fell 2% or more in four hours”, against what happened next, and passes the conditions that hold up statistically to the reflection step. It is the only route by which anything about the market reaches the loop. If the scan cannot see a pattern, no language model downstream can learn it. PE-2’s day-one success was this scan at work: the planted trigger was one of its 38 conditions.
The discovery test asks whether the scan finds anything it was not given. Each of 150 synthetic price histories (“tapes”, 4,800 hourly bars each) carries one pattern from one of five families:
- A is exactly one of the library’s conditions. It is the positive control (a case the test must catch, to prove the test itself works).
- B needs two conditions at once, a drop and a high RSI (a common momentum indicator), where neither alone predicts anything.
- C is triggered by trading volume, and the library has no volume condition.
- D fires only for a drop within a band (between 1% and 2%), while the library’s conditions are one-sided.
- E has nothing planted, to count false alarms.
Each family ran at three strengths, calibrated on family A alone so that A is found about every time, half the time and one time in ten, with 10 seeds (random variations of the same tape) each.
| Row | Strong | Medium | Weak |
|---|---|---|---|
| A: one library condition (control) | 10 of 10 | 8 of 10 | 1 of 10 |
| B: two conditions at once | 0 of 10 | 0 of 10 | 0 of 10 |
| C: a volume trigger | 0 of 10 | 0 of 10 | 0 of 10 |
| D: a drop within a band | 0 of 10 | 0 of 10 | 0 of 10 |
| E: nothing planted (false alarms) | 0 of 10 | 0 of 10 | 0 of 10 |
Family A was found in 10, 8 and 1 of 10 tapes. Families B, C and D were found in none of 90, and family E produced no false alarms. Under the rule registered that morning, this is the verdict LOOKUP: the loop retrieves patterns from its list; it does not discover them. The pre-registration also ruled out the obvious fix: extending the list of conditions until the missing families are caught. That would make the test pass by telling the loop the answer, and the loop would still find only what it was given.
What this does not show
It does not show that markets are efficient, or that language models cannot trade. It shows that this loop, as built, never demonstrated edge, and that its route from market data to new rules can only retrieve what was put into it. A follow-up design was red-teamed and judged unsound as written, partly because its main test amounted to widening the library; the arena was archived on 4 October 2026 instead of amending it.
For anyone building a self-improving agent, two things carry over. Test the loop on a pattern you planted before trusting it on real data, and plant one that lies outside what the loop was given; a loop that finds only its own library’s patterns is a lookup, however much language surrounds it. And compare against the simplest baseline for the whole window: here, holding the market beat every arm in every phase.
Method
How the work was done, and what went wrong along the way.
The arena ran from December 2025 to October 2026 on one machine with paper money. Phase 1 used Gemma 4 31B and phase 2 GLM-5.3 Flash, both through a flat-rate hosted service. Earlier runs with other models are not covered: a deploy in June 2026 reset the database, so the records on this page start on 2026-06-13.
Phase 1 had no pre-registration of its own: it was judged by the arena’s edge report against the index fund. Every later test had a dated pre-registration with its pass rule. Git confirms that the documents for phase 2, PE-2, Q1 and the discovery test were committed before their data existed. PE-1’s document is dated before its run but was first committed together with its results.
Known problems:
- A unit bug silently rejected between 12% and 35% of each advanced agent’s trade openings in phase 1 (16 of 136 to 46 of 133 in a 30-day audit). In PE-1, only 5 of 28 intended trades on the planted pattern were executed, so PE-1’s failure to trade the pattern partly reflects the bug; its failure to describe the pattern does not.
- In phase 2 the agent with the long skills block timed out on 84 of 514 decisions, against 30 for its twin, and each timeout counted as a hold.
- From 6 to 13 September, an ungated code change by the code-generation step altered the agent with skills in the 62-day comparison without passing the replay gate. It was found and reverted at the end of phase 1; the +5.85% includes those six days.
- PE-2’s “named the pattern” criterion was scored by a heuristic. The human check the pre-registration called for was never recorded.
- Q1 was underpowered: on a 144-day holdout, a strategy with a genuinely good Sharpe of 1.5 to 2 would also have failed its interval criterion. The default setting’s −30.70% does not depend on that.
- An audit on 2 July 2026 found two accounting bugs that had inflated equity (unrealised liquidation losses and missing entry fees) and withdrew every edge claim made before it.
Not measured: real money, other markets, other loop designs. The repository and database are private; the figures on this page were rechecked against both on 10 October 2026.