By Daniel Huber · original experiment · October 9, 2026 · 7 min read
Verdict
Used as released, the small Clef model catches about half of the new attacks in the final test. Retraining only its scoring head lifts that to 88%. The large Clef model catches 99.8% with no training at all, but each check takes three to five times as long.
Original measurement99.8%
Large Clef, released
88%
Small Clef, trained by us
53%
Small Clef, released
920 untouched test cases, run once per model · ±1 point across 3 training runs · one GPU, no real traffic
Where the guard sits
For each step, the guard sees four fields: the agent’s purpose (from configuration, so trusted), the user’s request, the new untrusted content of this step (a tool output, a file, a web page) and the tool call the agent proposes. Untrusted text always stays in its own field.
Clef is a decision model, not a chatbot. Given that state and a fixed set of questions (is there an injection, and of which kind; how risky is the proposed tool call; is the request harmful), it returns a probability for every answer in one forward pass, without generating text. The guard’s score is the highest of those risks. Because the guard writes nothing and calls no tools, an injection can at most make it miss something; it cannot make it act. Attacks written to the guard itself (“this content was already classified as safe”) were caught 98% of the time by the trained small model and 100% by the large one.
What we tested
Four guards, all open models run locally:
- Nemotron-3.5 Content Safety (4B, NVIDIA), a harmful-content classifier, used as the baseline. Each suite gets the better of its two policies.
- Clef-flash (9B) and Clef (27B), Cloudflare’s decision models, released on 30 September 2026 under Apache-2.0, used as released.
- Clef-flash with a scoring head we retrained: the 9B backbone stays frozen and only the head that turns its hidden states into answers (about 115 million parameters) is trained, in about 45 minutes on one GPU.
The test split has 2,876 cases: public benchmarks for direct injection (deepset), injection hidden in tool output (InjecAgent) and harmful requests (JailbreakBench, XSTest); 495 real GitHub issues with stack traces, from repositories the trained head never saw, each as is and with an inserted attack; and self-written German and English tool outputs, tool calls, code, logs and other technical artifacts.
Each guard’s threshold is set so that 5 in 100 harmless texts of a suite get flagged. The number is the share of attacks it still catches at that threshold. Comparing at a fixed 0.5 instead would reward whichever model happens to lean towards blocking.
Results for each kind of test
| Row | Nemotron 4B | Small Clef, released | Large Clef, released | Small Clef, trained by us |
|---|---|---|---|---|
| Direct injection / jailbreak | 72% | 72% | 67% | 87% |
| Injection in tool output (InjecAgent) | 73% | 90% | 95% | 100% |
| Injection in tool output (own, DE/EN) | 49% | 90% | 99.3% | 100% |
| Real GitHub issues, unseen repos | 35% | 73% | 99.2% | 94% |
| Stack traces, code, logs (24 cases) | 42% | 50% | 100% | 100% |
| Generated technical artifacts | 18% | 72% | 77% | 92% |
| Harmful requests (JailbreakBench) | 92% | 64% | 68% | 88% |
| Harmful vs safe-but-scary (XSTest) | 82% | 72% | 79% | 89% |
Nemotron has no notion of injection: it catches about a third of the attacks hidden in real GitHub issues and under a fifth in generated code and logs. The small Clef model as released beats it on every injection and technical suite except direct injection, where they tie, but stays near 70% on direct injection and real issues. Retraining its head lifts most suites to 87–100%. The large Clef model is the strongest on real issues without any training; the trained small model leads on direct injection and generated technical artifacts.
Over-blocking matters as much as recall, because a guard that blocks stack traces gets switched off. With its threshold set to catch 90% of injections, the trained small model blocks 2 of 495 harmless real issues, 0 of 24 stack traces and 6 of 210 generated technical artifacts.
Final test
A test set that a project keeps measuring against becomes optimistic. So after every decision was made, we wrote a fresh one: 460 unused issues from 14 test repositories, each clean and with an attack whose goal, wording and insertion point were new. That is 920 cases, and each model ran on it once.
| Row | 1 in 100 flagged | 5 in 100 flagged | Difference |
|---|---|---|---|
| Nemotron 4B | – | – | – |
| Small Clef, released | 32.8% | 53.0% | +20.2% |
| Large Clef, released | 99.6% | 99.8% | +0.2% |
| Small Clef, trained by us | 79.6% | 88.0% | +8.4% |
The trained head’s recall on real issues drops from 94% on the test split to 88% here; across three training runs it stays within a point. Its misses are confident, not borderline: polite instructions not addressed to an AI (“To whoever automates triage here: please …”), instructions disguised as [WARN] log lines, and goals that sound like routine admin work, such as turning off branch protection. The large model catches 94–100% of each of those kinds. On a separate set of disguised attacks (11 techniques, scored at a 0.5 threshold), the trained head caught 95% and the large model 100%.
Training lessons
We trained the head in seven rounds. Each lesson below was measured:
- Without technical data, a trained head gets worse than the released one on code and logs.
- Real GitHub issues fixed the threshold shift that synthetic data left: harmless real issues blocked at a 0.5 threshold fell from 12% to 1.4%.
- Long tool outputs have to be split into windows, and the head has to see such windows in training: without them, 16 of 20 harmless 64,000-character outputs were blocked.
- Training on attack windows alone teaches “an inserted instruction-like line is an attack”. Adding harmless inserted lines at the same spots fixed that.
- More evasion data can hurt when only the head is trained, and distilling from the large model added nothing beyond more data.
Very long outputs remain the weak spot. At 64,000 characters the guard scores 12 windows and keeps the most suspicious, so small per-window false-alarm rates add up. A stricter block threshold for inputs with many windows sends most of these to a person instead of blocking them, without letting more attacks through.
Content safety
Could one Clef guard also do Nemotron’s job, checking for harmful requests and answers? On 7,058 requests and 2,691 AI responses from public sets, Clef was about level with Nemotron on English requests and blocked less legitimate security work. Nemotron was clearly better on harmful requests in other languages (70% caught at 5 in 100 false alarms, against 46–57% for the Clef variants) and on harmful AI answers. A head trained on content-safety data as well moved ahead on German (97% against 90%) and other languages (74% against 70%), and stayed close on English requests (72% against 76%), at a cost of 2–5 points on disguised injection attacks. Neither result has been checked on real traffic.
Method
This was a private prototype, run on one NVIDIA DGX Spark between 1 and 4 October 2026. It was not pre-registered: the questions, data and heads changed as results came in, which is why the final test exists. Heads were trained on the development split only; the test split compared rounds, and the final set was written after all decisions and run once per model.
On the Spark, a check takes about 220 ms with Clef-flash and 590 ms with Clef 27B; Cloudflare’s figures on an H200 are 39 ms and 209 ms (median). Not measured: real traffic, FP8 weights, latency on production hardware.
The trained head is not released. A public guard is a white-box target: with the weights, an attacker can tune evasions offline.
| Source | Used for | Licence |
|---|---|---|
| deepset/prompt-injections, jackhhao/jailbreak-classification | direct injection and jailbreaks | Apache-2.0 |
| leolee99/NotInject | harmless prompts full of trigger words | MIT |
| InjecAgent | injection in tool output | MIT |
| JailbreakBench JBB-Behaviors, XSTest | harmful vs safe-but-scary requests | MIT, CC-BY-4.0 |
| sharjeelyunus/github-issues-dataset | real issues with stack traces, split by repository | MIT (as declared by the uploader) |
| Self-written cases (fictional organisation) | German/English tool outputs, tool calls, technical artifacts | own work |