Skip to content

Benchmark

flowchart LR
  T["Same spec: sitegen, 9 modules"] --> A["Arm 1: superpowers 6.4.1"]
  T --> B["Arm 2: superlite 0.1.0"]
  A --> H["Hidden acceptance suite: 89 tests"]
  B --> H
  H --> R["Compare time, cost, tests, turns"]

Both plugins built sitegen, a stdlib Python site generator, from bench/task/SPEC.md (9 modules). Each ran headless on Opus. The hidden suite checks the spec only, and neither arm saw it. The run was on 2026-10-16 and corrected on 2026-10-17. The correction fixed a bug that summed every result event and inflated costs 3x and 9x.

Metric superpowers 6.4.1 superlite 0.1.0
Wall-clock 34.4 min 8.8 min (3.9x faster)
Cost (API-equivalent) $8.15 $3.28 (60% cheaper)
Opus / Sonnet / Haiku cost $6.43 / $1.72 / none $1.53 / $1.75 / $0.002
Hidden acceptance tests 89 / 89 89 / 89
Own tests written 298 208
Stops-for-human 0 0
Coordinator turns 68 46
Context per request (median / max) 108k / 154k 70k / 90k
Subagent dispatches 21, strictly serial 10, up to 5 concurrent
Agent-busy wall time 25.1 min 4.8 min

Start with the hidden tests. Both arms passed 89 of 89, so the time and cost gaps do not come from a worse result. Wall-clock is the time you wait. Cost is API-equivalent, meaning what the tokens would cost at list price; on a subscription it appears as usage-limit use instead.

Then read the causes. Parallel waves drive the speed: the serial chain fell from 21 dispatches to 3 levels. Superpowers’ review after every task took about 12 minutes. Planning took about one minute in both arms. Cost follows context size: superlite’s Opus read 2.3M cached tokens against 8.7M, because it took fewer turns at a smaller context.

The README reports a second task, a site generator plus a spreadsheet engine (177 hidden tests). The 16.1 minute figure is one run of the current build.

What we measured superpowers superlite
Wall-clock to a finished build 33.6 min (stock, 3 runs: 21.2 / 31.9 / 47.6); 23.5 min (orchestrated, 3 runs) 16.1 min
Fastest-to-slowest spread, 3 runs 26.4 min 9.1 min
Hidden tests passed 177 / 177 177 / 177
  • There is one run per arm in the first benchmark, so timings could move by 30%.
  • The arms overlapped in time and shared rate limits.
  • The hidden suite checks the spec only, so it does not measure review quality.
  • On the later task, superlite can cost more than superpowers and scores lower on the extra robustness probes (4 of 7 against about 5.7 of 7).

Example: reading the results in five checks

Section titled “Example: reading the results in five checks”

Say you opened this page to decide whether the claim “3.9x faster, 60% cheaper” is worth trusting.

  1. Check the score first. Find “Hidden acceptance tests” in the first table.

    Hidden acceptance tests
    superpowers 6.4.1 89 / 89
    superlite 0.1.0 89 / 89

    Both arms passed the same hidden suite, which neither had seen. So the speed and cost gaps do not come from a weaker result.

  2. Read the time, then its cause. 34.4 min against 8.8 min is what you wait. The rows below explain it: 21 dispatches strictly serial against 10 with up to 5 at once, and 25.1 min of agent-busy time against 4.8.

    superlite builds independent tasks in parallel waves, which is the whole speed gain. See Parallel build.

  3. Read the cost split. $8.15 against $3.28 is API-equivalent, meaning list price. On a subscription it shows up as usage-limit use instead. The split row shows Opus spend falling from $6.43 to $1.53, because superlite takes fewer turns at a smaller context: median 70k per request against 108k.

  4. Check how noisy it is. The first table is one run per arm, so timings could move by 30%. The later task ran superpowers three times, and its slowest run was 26.4 min slower than its fastest. superlite’s spread was 9.1 min.

  5. Check what the test misses. The hidden suite checks the spec only, so it says nothing about review quality. On the later task superlite scored 4 of 7 on the extra robustness probes against about 5.7 of 7.

You decide: whether a faster, cheaper build with weaker robustness probes fits your work. For a prototype it may; for a payment path you may want the deeper review.

What you end up with: a verdict you can defend. Same hidden score, a large time gap with a known cause, a smaller cost gap with a known cause, and two limits (one run, spec-only checks). The harness is in bench/ and the full write-up is docs/benchmark-report.md.