Benchmark
The setup the research produced changed the cost, not the result.
The research and the repository reviews ended in one setup. We tested it against stock Claude Code under a pre-registered design. Six results, in the order they matter to someone choosing a configuration.
What was tested
The setup combined the pieces the reviews rated highest with rules written from the research: a short CLAUDE.md of working preferences; the mattpocock/skills skills grilling, grill-me and diagnosing-bugs as shipped and tdd and to-tickets adapted; the Codex review wrapper from shimo4228/codex-review; six skills of our own for contracts, plans, decision records and review; three review subagents; and four hooks, among them the forced skill-evaluation hook from spences10/svelte-claude-skills. It was measured as a bundle.
1. The setup does not hurt pass rate
The harm check ran stock Claude Code and the setup on the same 69 build tasks, with the same model (Claude Opus 5.5 at medium effort), the same pinned CLI (2.1.285) and isolated configuration directories. Each task's score is the share of its hidden acceptance tests that pass on a fresh copy of the agent's final workspace. Tasks where the two disagreed by 0.10 or more were run a second time, as registered.
| Stock Claude Code | Setup | Difference | |
|---|---|---|---|
| Mean share of hidden tests passed | 89.8% | 89.4% | −0.4 pts |
| One-sided 95% lower bound of the difference | −2.0 pts | ||
| Tasks better / equal / worse with the setup | 2 / 60 / 7 |
2. It costs about a quarter more
Cost at API prices, tokens, wall-clock time and turns were read from every stored transcript. The figure shows the median ratio per task with a bootstrap 95% interval. The likely sources are the setup's standing context (about 2,800 tokens of instructions and skill descriptions loaded in every session) and the forced skill-evaluation hook. The original hook cost ×1.88 output tokens per prompt in our activation test; the reworded version in the setup cost ×1.03 tokens and ×1.09 cost. The study did not separate the sources.
3. No gain where there was room for one
Most tasks were solved by both, so a gain could only show where there was room for one. Before the harm check ran, 14 tasks were picked as the headroom set from earlier screens with the setup's CLAUDE.md, where they passed between 10% and 95% of hidden tests. None moved by more than 0.10. One task outside the set with room to improve, openai-go-a29 (stock 0.60), also showed no difference. The set is small and noisy, and we claim no more from it than the absence of a visible effect.
| Task | Stock | Setup | Difference |
|---|---|---|---|
| Lightning-AI__pytorch-lightning-eea74433ce | 0.75 | 0.69 | −0.06 |
| stanfordnlp__dspy-33aaa19e03 | 0.88 | 0.81 | −0.06 |
| pypa__packaging-58c6cd700b | 0.89 | 0.83 | −0.06 |
| tobymao__sqlglot-da9a9c9664 | 0.67 | 0.64 | −0.03 |
| h3js__srvx-6b3b2910ae | 0.89 | 0.89 | 0.00 |
| mswjs__interceptors-7f1b37ca80 | 0.67 | 0.67 | 0.00 |
| openai__openai-go-7e4e1b1912 | 0.50 | 0.50 | 0.00 |
| pypa__packaging-23669b1b69 | 0.50 | 0.50 | 0.00 |
| slice-py-cli-prefix-policy-forge | 0.95 | 0.95 | 0.00 |
| stanfordnlp__dspy-d58c3007eb | 0.33 | 0.33 | 0.00 |
| tobymao__sqlglot-e3ed74860f | 0.50 | 0.50 | 0.00 |
| tox-dev__filelock-126cab7456 | 0.74 | 0.74 | 0.00 |
| tox-dev__platformdirs-dfaeabf921 | 0.86 | 0.86 | 0.00 |
| stanfordnlp__dspy-e8ec009c6b | 0.24 | 0.34 | +0.10 |
4. A forced-evaluation hook changes behaviour, measurably
The hook from spences10/svelte-claude-skills prints an instruction before every prompt that tells the model to evaluate each installed skill and load the relevant ones. On 41 prompts that call for a specific skill, it raised correct activation from 62% to 93% (+30.5 points, 95% CI 17.1 to 43.9). It made no false activations on 5 near-miss prompts (a small sample), though it also loaded extra skills on 14 of the 41 prompts. Re-tested with the setup's own skills and wording, the gain was +21.2 points at the interim look, on half the planned prompts (99% CI 6.1 to 39.4). This is the one configuration effect the study resolved. It is a change in process: the setup that contains the hook built no better than stock.
5. Frontier models solve most of what you can build to test them
A comparison can only show a gain on tasks the baseline fails. We spent much of the study trying to build such tasks and mostly failed. The figure shows how Claude Opus 5.5 did on each pool, with the setup's CLAUDE.md.
| Pool | Solved | Partly | Not solved | Tasks |
|---|---|---|---|---|
| Bug-finding: 40 known bugs in real commits (TB-R) | 27 | 8 | 5 | 40 |
| Build: 36 features and greenfield slices (TB-B v6) | 30 | 4 | 2 | 36 |
| Build: 15 full-day greenfield slices | 15 | 0 | 0 | 15 |
| Build: 6 larger real-repository features | 3 | 3 | 0 | 6 |
| Build: 8 ML/DL features and long reruns | 1 | 6 | 1 | 8 |
- Precise specifications make tasks easy. Our "full-day" greenfield projects (targeted at 800 to 2,000 lines of reference code, 35 to 50 hidden tests, specifications audited by a second model so that every test was determined by the spec) were finished in one to six minutes each. 147 additional adversarial tests, written to target edge cases, failed the stored solutions once.
- The model could not tell what was hard for it. In one probe, asked to predict which of 57 already-measured tasks Opus 5.5 would fail, Opus 5.5 flagged 7 of the 9 that were hard and 26 of the 48 that were easy: 21% precision against a 16% base rate.
- Difficulty lives in other people's code. The tasks the model failed were features re-implemented inside real repositories from their pull-request descriptions, especially in machine-learning libraries (dspy features scored 0.23 and 0.33). Only 7 of 124 candidate features from 22 ML/DL libraries could even be tested on a laptop.
6. The instrument is part of the result
Several faults in our own harness produced results that looked like findings before they were caught. A scoring crash recorded every agent in one trial as failing. A task prompt that began with a hyphen was read as a command-line flag. A verifier that timed out under parallel load turned a 0.83 into a 0, and because of that one task the harm check's first pass read as harm overall (bound −5.8). Re-scoring every stored workspace caught it. The tooling page lists the faults that mattered most. Anyone running their own comparison should expect the same kinds of error and budget for re-verification.
What this does not show
- One model (Claude Opus 5.5) at one effort level, one CLI version (2.1.285), one machine.
- Build tasks of a few hours at most. Multi-day projects, long sessions with many prompts, and team workflows were not measured.
- The setup was tested as a bundle. A single component could help while another hurts.
- Of the 107 configurations we specified for testing, one, the forced-evaluation hook, was measured on its own, in two versions. Most repositories were read, not benchmarked. The arms page lists every configuration and what happened to it.