topbot

Method

How the study was run, and how its design changed.

The design of each benchmark campaign was fixed before its results were seen and changed only by dated, recorded decisions. This page states the rules, then walks through each change and the evidence that caused it.

Rules that held throughout

Read from source
Every repository was cloned at a fixed commit and read file by file. Every web source was stored with its address, retrieval time and a content hash, and every claim in the research was checked against the stored text.
Pre-registration
Every campaign's hypothesis, arms, tasks, metric, decision rule and stopping rule were committed to version control before its first measured run. Later changes were recorded as dated lines rather than rewrites.
Paired design
Arms differ only in configuration. Same tasks, same model and effort, same pinned CLI binary (2.1.285), each run in its own isolated configuration directory. Each arm's configuration is hashed and the hash recorded with every run.
Calibration first
Before a comparison on the review bank or the first build bank, the control arm ran the whole bank twice. Tasks the control always solves or always fails cannot show a difference, so the comparison window was set to tasks the control solves between 25% and 85% of the time. Later screens kept tasks scored between 0.10 and 0.95. The harm-check bank was deliberately left near ceiling, since a drop on a solved task is what a harm check looks for.
Trial first
The skill-activation campaigns and the build calibration started with a trial of three tasks; if an arm failed to start, telemetry was missing or a hook stayed silent, nothing else ran.
Hidden tests
Build tasks are scored by hidden acceptance tests run on a fresh copy of the agent's final workspace, as the share of held-out tests passing. Separate checks record whether visible tests were modified, which files outside the allowed set changed, and whether existing tests regressed.
Contamination control
The fixes and features in real-repository tasks were committed after the model's knowledge cutoff. For the review bank and the first build bank, a recall probe asked the model to name what a change would add without seeing it; items it could recall were removed.
Interim looks with a stricter bar
Campaigns could stop at half the planned runs only if the result held at 99% confidence; the full run used 95%.
Independent models for judgement
Labels, grades and drafted tasks came from models of other families than the agent under test; harvested tasks come from real commits. From 11 October each outside model was called only through its own maker's API.
Re-verify
For the harm check, every stored workspace was scored again on its own after the runs. A score that changes on re-verification is a harness fault and is corrected by a registered rule. One of 148 was.
Stop rules in advance
Each campaign's budget and stopping rule were registered before its first run, including the harm check's rule that closed the benchmark. The rule that ended the search for hard tasks was written after the first screening round.

How the design evolved

  1. Research

    34 research documents synthesised Anthropic's documentation and system cards, practitioner essays, threads and papers on configuring Claude Code, from 1,040 sources. 46 public repositories of skills, hooks, agents and plugins were reviewed from source, each with a written verdict. Claude Code's own system prompts were captured for every model, so that no rule in the setup would repeat what the harness already tells the model.

  2. A ranked plan of 107 configurations

    Every candidate change became an "arm" with a hypothesis, control, metric, predicted effect and falsifier, ranked by what a result would settle, the chance of resolving it, and its cost. A purpose-built runner executes Claude Code headless against task banks. The first campaign measured the forced skill-evaluation hook: +30.5 points in correct skill activation.

  3. Spending discipline

    A cost cut kept only the 8 arms with the best expected insight per unit of usage. Interim looks at half the planned runs were added. The review-task bank was rebuilt from the commits that introduced real bugs, with the fix's own tests as the oracle.

  4. The setup is built

    The research and the first result were turned into a working configuration: a CLAUDE.md, 12 skills, 3 subagents, 4 hooks and settings. It became the control for later arms.

  5. The review bank hits a ceiling

    Re-calibrated on the built setup, Opus 5.5 found 27 of 40 known bugs on every attempt; only 8 sat in the window where a difference could show. The setup's own version of the hook was confirmed (+21.2 points).

  6. Build first, and the search for hard tasks

    The cost cut was removed and the review campaign dropped: the question that mattered was whether a setup helps build software. The build bank's calibration showed the same ceiling (31 of 37 tasks above 0.85). Three attempts to make harder tasks followed: full-day greenfield projects (solved in minutes), adversarial tests (146 of 147 passed by the stored solutions) and larger features from real repositories, including 22 ML/DL libraries. A model asked to predict difficulty could not. A stop rule capped the search at two screening rounds.

  7. A harm check, then stop

    Since gains could not be resolved on tasks the model already solves, the goal became narrower and answerable: show that the setup does not make stock Claude Code worse, and measure what it costs. The harm check found no harm and a 25% higher cost. The benchmark closed as registered.

Why the narrower question is still worth answering

A drop from a task the model always solves is easy to see, so a harm check needs far fewer runs than a search for gains. The answer is useful on its own terms: a carefully assembled setup did not make things worse, and it did make them more expensive.