Original research
Experiments
Original experiments on questions people building agents face, including an LLM trading benchmark. Every result comes with its method and how sure we are.
One question each, measured and reported with its uncertainty. Pre-registered experiments write down what we expect before running anything.
- Agentic RAG Partial results
Tree navigation vs vector search on long and look-alike documents
A pre-registered comparison of PageIndex-style tree navigation, RAPTOR, tuned hybrid vector search and whole-document reading on 847 questions over structured-product documents, long prospectuses and FinanceBench.
+15 pts PageIndex vs tuned hybrid search, all questions
- Learning & Adaptation Complete
Can a self-improving trading agent find a pattern it was not told about?
Ten months of LLM trading agents wrapped in a self-improvement loop on crypto futures with paper money: no configuration beat a passive index fund, and a final pre-registered test showed the loop only finds patterns already in its own library.
0 of 90 planted patterns outside the loop’s library found
- Evaluation & Metrics Complete
Reading a model's mistakes from its middle layer
A pre-registered test of whether two Jacobian lenses on layer 24 of Gemma 4 12B flag errors planted in the model's own answers better than the model's output probabilities do.
18 of 30 planted errors caught by the J-lens on layer 24 (chance is 15); the model’s own output caught 20
- Safety & Guardrails Complete
A small decision model as a prompt-injection guard
Open models compared as prompt-injection guards for a coding agent at a fixed false-alarm rate: Cloudflare's Clef models with and without a retrained scoring head, and NVIDIA's Nemotron.
99.8% of new attacks caught by the large Clef model, untrained
- Benchmark Complete
LLM trading decisions
Which language models make the best crypto-futures trading calls, and under what market conditions? A deterministic backtest of agentic and single-call LLM traders across 20 historical scenarios, against buy-and-hold, technical-analysis and random baselines.
540 backtest runs: 5 models, 20 scenarios