topbot

A measurement of Claude Code setups · September 2026 to October 2026

A month of building and benchmarking a Claude Code setup found no measurable gain in build results, and a 25% higher estimated cost.

We researched how people configure Claude Code, assembled the most defensible setup we could, part third-party (skills and a hook from public repositories), part our own rules, skills and subagents (some adapted from other public sources), and then tested it, as a bundle, against stock Claude Code on real build tasks. It did not make the agent worse. It did not make it better. Its estimated cost per task was about a quarter higher.

−0.4 ptspass rate, setup minus stock, on 69 build tasks (one-sided 95% bound −2.0)
×1.25estimated cost per task with the setup, at API prices (95% CI ×1.21 to ×1.29)
1,037benchmark runs of Claude Code, 260 million tokens
3.2 billiontokens processed by the 17 Claude Code sessions run in this project's directory (a lower bound)

What we asked

Claude Code ships with defaults. A large ecosystem of third-party configurations promises to improve on them: skills, slash commands, hooks that inject instructions into every prompt, review subagents, long CLAUDE.md files. We wanted to know which of these make the agent produce better software, for a single developer building web, app and command-line projects with the latest Claude model.

What we found

  • No harm. The full setup scored 89.4% of hidden tests against stock Claude Code's 89.8%. On 60 of 69 tasks the two were identical. The registered margin for "no harm" was 5 points; the bound was −2.0.
  • No measurable gain. On 14 tasks picked in advance for having room to improve, the setup moved no task by more than 0.10.
  • A real cost. At API prices, the setup used ×1.27 the tokens, ×1.21 the wall-clock time and ×1.25 the money per task.
  • One component did measurably change behaviour. A hook that forces the model to evaluate its installed skills on every prompt raised correct skill loading by 21 to 31 points. That change in process did not show up as better build results.
  • The model was hard to challenge. Claude Opus 5.5 solved most of the tasks we could construct. Of 16 full-day greenfield projects screened, with 35 to 50 hidden tests each (one excluded for a test-harness defect), it passed at least 95% of the tests on all 15, and all of them on 13. Measuring an improvement needs tasks it fails; those turned out to be rare and expensive to find.

The findings page has the numbers and figures. The method page explains how the study was designed and how the design changed as results came in. The effort page counts what went into it, and the inventory lists everything we read, analysed, specified and ran.

Who should care

Anyone deciding whether to install a popular Claude Code configuration. On this evidence, the default setup is a strong baseline, a curated setup costs more without improving results on tasks like these, and any claimed gain deserves a measurement that includes the cost.