PirateBench
Agents build slices of our real projects from an empty workspace; a rubric grades them. Scores are out of 100, after intervention penalties. Every run of the two newest scored versions is listed, failures included; older and calibration epochs are summarized. 167 runs. Claude Code runs before v3 saw our notes file (no answers).
Build tiersscored v3
Tiers use the worst complete seed try: planning / inline 95+, quality worker 75+, budget worker 50+, otherwise not for building; correct = 95+, with cost and time across all complete tries per correct build at list prices.
| model | tier | worst try | $/correct build | time/correct build | cache share |
|---|---|---|---|---|---|
| gpt-6.1-sol | planning / inline | 98.9 | $0.57 | 25.9 min | 48% |
| opus-5-5 | planning / inline | 95.4 | $3.71 | 29.8 min | 95% |
| gpt-6-sol | quality worker | 80.4 | none correct; $0.66/build | – | 48% |
| gpt-6-luna | not for building | 45.9 | none correct; $0.02/build | – | 47% |
| haiku-4-5 | not for building | 3.2 | none correct; $1.08/build | – | 98% |
Leaderboardscored v3
Out of 100 on the Kalshi data collection, T slices, items K01–K40, T01–T36; scored runs, rubric v3
epoch 68781f26be8f-de803160205e-0f6ff1c49115-7698d52d5555
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| gpt-6.1-sol | 6 | 98.9 | 99.0 | 97.9–100.0 | $0.00 list $0.29 |
| opus-5-5 | 6 | 96.5 | 96.5 | 94.3–100.0 | $0.00 list $1.86 |
| gpt-6-sol | 6 | 91.6 | 88.7 | 72.8–96.5 | $0.00 list $0.33 |
| gpt-6-luna | 6 | 65.2 | 63.0 | 37.9–85.1 | $0.00 list $0.01 |
| haiku-4-5 | 6 | 4.3 | 5.2 | 2.0–19.2 | $0.00 list $0.54 |
Score = each slice's median run, weighted by its items (K 40, T 36).
Score per slice
All runs (30)
| 2026 date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 10-02 20:10 | haiku-4-5 | T | 0 | 2.0 | 1/35/0/36 | |
| 10-02 20:01 | haiku-4-5 | K | 0 | 4.2 | 2/38/0/40 | |
| 09-30 16:21 | haiku-4-5 | T | 2 | 2.0 | 1/35/0/36 | |
| 09-30 16:13 | haiku-4-5 | T | 1 | 19.2 | 7/28/1/36 | |
| 09-30 15:53 | opus-5-5 | T | 2 | 96.5 | 35/1/0/36 | |
| 09-30 15:45 | opus-5-5 | T | 1 | 96.5 | 35/1/0/36 | |
| 09-30 15:37 | haiku-4-5 | K | 1 | 6.3 | 3/37/0/40 | |
| 09-30 15:25 | gpt-6-sol | K | 2 | 90.2 | 36/4/0/40 | |
| 09-30 15:14 | gpt-6.1-sol | T | 2 | 100.0 | 36/0/0/36 | |
| 09-30 15:06 | opus-5-5 | T | 0 | 96.5 | 35/1/0/36 | |
| 09-30 14:58 | gpt-6-sol | K | 1 | 87.2 | 36/3/1/40 | |
| 09-30 14:47 | gpt-6-luna | T | 2 | 54.1 | 20/16/0/36 | |
| 09-30 14:34 | haiku-4-5 | K | 2 | 6.3 | 3/37/0/40 | |
| 09-30 14:22 | gpt-6-sol | K | 0 | 77.4 | 32/7/1/40 | |
| 09-30 14:12 | gpt-6-luna | T | 1 | 54.7 | 21/15/0/36 | |
| 09-30 13:53 | gpt-6-luna | K | 2 | 85.1 | 35/5/0/40 | |
| 09-30 13:42 | gpt-6-luna | T | 0 | 71.4 | 27/9/0/36 | |
| 09-30 13:34 | opus-5-5 | K | 0 | 96.4 | 39/1/0/40 | |
| 09-30 13:23 | gpt-6-luna | K | 1 | 37.9 | 16/24/0/40 | |
| 09-30 13:12 | gpt-6.1-sol | T | 0 | 100.0 | 36/0/0/36 | |
| 09-30 13:04 | opus-5-5 | K | 1 | 94.3 | 38/2/0/40 | |
| 09-30 12:53 | gpt-6-luna | K | 0 | 74.7 | 31/8/1/40 | |
| 09-30 12:42 | gpt-6.1-sol | T | 1 | 100.0 | 36/0/0/36 | |
| 09-30 12:34 | gpt-6-sol | T | 2 | 96.5 | 35/1/0/36 | |
| 09-30 12:26 | opus-5-5 | K | 2 | 100.0 | 40/0/0/40 | |
| 09-30 12:15 | gpt-6.1-sol | K | 2 | 97.9 | 39/1/0/40 | |
| 09-30 12:04 | gpt-6-sol | T | 1 | 72.8 | 27/9/0/36 | |
| 09-30 11:56 | gpt-6.1-sol | K | 1 | 97.9 | 39/1/0/40 | |
| 09-30 11:45 | gpt-6-sol | T | 0 | 96.5 | 35/1/0/36 | |
| 09-30 11:37 | gpt-6.1-sol | K | 0 | 97.9 | 39/1/0/40 |
Leaderboardscored v2
Out of 100 on the Kalshi data collection, MathForge server core, Recipe Kroger cart slices, items K01–K40, M01–M30, R01–R30; scored runs, rubric v2
epoch bc153d1e05b8-cfd29c413b22-7c295401355c-aba3dbce9d85
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| opus-5-5 | 9 (9 regraded) | 91.2 | 90.2 | 78.0–100.0 | $0.00 list $2.35 |
| gpt-6-sol | 9 (9 regraded) | 81.8 | 92.3 | 46.9–97.1 | $0.00 list $0.41 |
| deepseek-v4-flash-0731 | 9 (9 regraded) | 62.8 | 90.2 | 4.2–95.1 | $0.11 |
| mimo-v2.6-flash | 3 (3 unscored) | – | – | – | $0.08 |
| mimo-v2.6-pro | 6 (6 unscored) | – | – | – | $0.18 |
Score = each slice's median run, weighted by its items (K 40, M 30, R 30).
Score per slice
All runs (63)
| 2026 date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 09-28 15:45 | opus-5-5 | R | 2 | 100.0 | 30/0/0/30 | |
| 09-28 15:45 | opus-5-5 | R | 1 | 95.1 | 29/1/0/30 | |
| 09-28 15:44 | opus-5-5 | R | 0 | 90.2 | 28/2/0/30 | |
| 09-28 15:44 | gpt-6-sol | R | 2 | 97.1 | 29/1/0/30 | |
| 09-28 15:44 | gpt-6-sol | R | 1 | 92.3 | 28/2/0/30 | |
| 09-28 15:44 | gpt-6-sol | R | 0 | 89.4 | 27/3/0/30 | |
| 09-28 15:43 | deepseek-v4-flash-0731 | R | 2 | 90.2 | 28/2/0/30 | |
| 09-28 15:43 | deepseek-v4-flash-0731 | R | 1 | 88.5 | 26/4/0/30 | |
| 09-28 15:43 | deepseek-v4-flash-0731 | R | 0 | 91.4 | 27/3/0/30 | |
| 09-28 15:43 | opus-5-5 | M | 2 | 100.0 | 30/0/0/30 | |
| 09-28 15:43 | opus-5-5 | M | 1 | 89.4 | 27/3/0/30 | |
| 09-28 15:42 | opus-5-5 | M | 0 | 100.0 | 30/0/0/30 | |
| 09-28 15:42 | gpt-6-sol | M | 2 | 95.1 | 29/1/0/30 | |
| 09-28 15:42 | gpt-6-sol | M | 1 | 95.1 | 29/1/0/30 | |
| 09-28 15:41 | gpt-6-sol | M | 0 | 95.1 | 29/1/0/30 | |
| 09-28 15:41 | deepseek-v4-flash-0731 | M | 2 | 95.1 | 29/1/0/30 | |
| 09-28 15:41 | deepseek-v4-flash-0731 | M | 1 | 94.6 | 28/1/1/30 | |
| 09-28 15:41 | deepseek-v4-flash-0731 | M | 0 | 94.8 | 28/1/1/30 | |
| 09-28 15:40 | opus-5-5 | K | 2 | 81.5 | 34/6/0/40 | |
| 09-28 15:36 | opus-5-5 | K | 1 | 81.5 | 34/6/0/40 | |
| 09-28 15:30 | opus-5-5 | K | 0 | 78.0 | 33/7/0/40 | |
| 09-28 15:26 | gpt-6-sol | K | 2 | 46.9 | 21/19/0/40 | |
| 09-28 15:21 | gpt-6-sol | K | 1 | 83.0 | 33/6/1/40 | |
| 09-28 15:17 | gpt-6-sol | K | 0 | 63.9 | 27/13/0/40 | |
| 09-28 15:12 | deepseek-v4-flash-0731 | K | 2 | 4.2 | 2/38/0/40 | |
| 09-28 15:07 | deepseek-v4-flash-0731 | K | 1 | 18.2 | 8/32/0/40 | |
| 09-28 15:03 | deepseek-v4-flash-0731 | K | 0 | 20.3 | 9/31/0/40 | |
| 09-28 10:20 | mimo-v2.6-pro | K | 2 | – | 28/11/1/40 | harness error: wrong route (OpenRouter, not Xiaomi direct) |
| 09-28 09:17 | mimo-v2.6-pro | K | 1 | – | 30/9/1/40 | timeout; harness error: wrong route (OpenRouter, not Xiaomi direct) |
| 09-28 08:12 | mimo-v2.6-pro | K | 0 | – | 31/8/1/40 | timeout; harness error: wrong route (OpenRouter, not Xiaomi direct) |
| 09-28 06:45 | mimo-v2.6-pro | K | 1 | – | 0/40/0/40 | harness error: provider rate limit: Provider rate limit reached. Please retry shortly. (request_id=req_28544fd59f16) |
| 09-28 06:41 | mimo-v2.6-pro | K | 0 | – | 0/40/0/40 | harness error: provider rate limit: Provider rate limit reached. Please retry shortly. (request_id=req_86de6842cc8d) |
| 09-28 06:39 | mimo-v2.6-pro | K | 2 | – | 0/40/0/40 | harness error: provider rate limit: Provider rate limit reached. Please retry shortly. (request_id=req_dc3769037d1e) |
| 09-28 03:30 | mimo-v2.6-flash | K | 0 | – | 0/40/0/40 | timeout; harness error: wrong route (OpenRouter, not Xiaomi direct) |
| 09-28 03:30 | mimo-v2.6-flash | K | 1 | – | 0/40/0/40 | timeout; harness error: wrong route (OpenRouter, not Xiaomi direct) |
| 09-28 03:30 | mimo-v2.6-flash | K | 2 | – | 0/40/0/40 | timeout; harness error: wrong route (OpenRouter, not Xiaomi direct) |
| 09-27 17:50 | opus-5-5 | R | 2 | – | 30/0/0/30 | not counted: busy machine (load1 30.2) |
| 09-27 17:50 | opus-5-5 | R | 1 | – | 29/1/0/30 | not counted: busy machine (load1 27.3) |
| 09-27 17:49 | opus-5-5 | R | 0 | – | 28/2/0/30 | not counted: busy machine (load1 28.0) |
| 09-27 17:49 | gpt-6-sol | R | 2 | – | 29/1/0/30 | not counted: busy machine (load1 25.3) |
| 09-27 17:49 | gpt-6-sol | R | 1 | – | 28/2/0/30 | not counted: busy machine (load1 21.4) |
| 09-27 17:48 | gpt-6-sol | R | 0 | – | 27/3/0/30 | not counted: busy machine (load1 21.2) |
| 09-27 17:48 | deepseek-v4-flash-0731 | R | 2 | – | 28/2/0/30 | not counted: busy machine (load1 18.8) |
| 09-27 17:48 | deepseek-v4-flash-0731 | R | 1 | – | 26/4/0/30 | not counted: busy machine (load1 18.2) |
| 09-27 17:47 | deepseek-v4-flash-0731 | R | 0 | – | 27/3/0/30 | not counted: busy machine (load1 18.9) |
| 09-27 17:47 | opus-5-5 | K | 2 | – | 34/6/0/40 | not counted: busy machine (load1 28.4) |
| 09-27 17:42 | opus-5-5 | K | 1 | – | 34/6/0/40 | not counted: busy machine (load1 24.5) |
| 09-27 17:36 | opus-5-5 | K | 0 | – | 30/9/1/40 | not counted: busy machine (load1 38.7) |
| 09-27 17:32 | gpt-6-sol | K | 2 | – | 20/20/0/40 | not counted: busy machine (load1 80.2) |
| 09-27 17:27 | gpt-6-sol | K | 1 | – | 26/14/0/40 | not counted: busy machine (load1 58.0) |
| 09-27 17:22 | gpt-6-sol | K | 0 | – | 27/13/0/40 | not counted: busy machine (load1 19.1) |
| 09-27 17:17 | gpt-6-sol | M | 1 | – | 29/1/0/30 | not counted: busy machine (load1 22.0) |
| 09-27 17:16 | gpt-6-sol | M | 2 | – | 29/1/0/30 | not counted: busy machine (load1 20.0) |
| 09-27 17:16 | deepseek-v4-flash-0731 | K | 2 | – | 2/38/0/40 | not counted: busy machine (load1 25.4) |
| 09-27 17:11 | gpt-6-sol | M | 0 | – | 29/1/0/30 | not counted: busy machine (load1 26.6) |
| 09-27 17:10 | deepseek-v4-flash-0731 | K | 1 | – | 8/32/0/40 | not counted: busy machine (load1 28.6) |
| 09-27 17:05 | deepseek-v4-flash-0731 | M | 2 | – | 29/1/0/30 | not counted: busy machine (load1 27.3) |
| 09-27 17:05 | opus-5-5 | M | 2 | – | 30/0/0/30 | not counted: busy machine (load1 25.0) |
| 09-27 17:04 | opus-5-5 | M | 1 | – | 27/3/0/30 | not counted: busy machine (load1 23.6) |
| 09-27 17:03 | deepseek-v4-flash-0731 | K | 0 | – | 9/31/0/40 | not counted: busy machine (load1 29.5) |
| 09-27 16:51 | opus-5-5 | M | 0 | – | 30/0/0/30 | not counted: busy machine (load1 30.7) |
| 09-27 16:46 | deepseek-v4-flash-0731 | M | 0 | – | 28/1/1/30 | not counted: busy machine (load1 34.1) |
| 09-27 16:40 | deepseek-v4-flash-0731 | M | 1 | – | 28/1/1/30 | not counted: busy machine (load1 41.7) |
Older and calibration epochs (5)
- scored v1
c4dbfc7e5c5a-644f71243d5d-c182c64574b7-88e7f43dcd6d· opus-5-5 49.9 · 24 runs - calibration
c4dbfc7e5c5a-644f71243d5d-c182c64574b7-88e7f43dcd6d· opus-5-5 · harness 2 75.0 · 20 runs - calibration
46fc803f1a9c-ec54f73daa8b-c182c64574b7· deepseek-v4.1-flash (run as deepseek-v4-flash-0731) 60.0 · 10 runs - calibration
fe823afec90b-7c33790e499b-c182c64574b7· deepseek-v4.1-flash (run as deepseek-v4-flash-0731) 65.0 · 9 runs - unknown
unknown· deepseek-v4-flash-0731 – · 11 runs