PirateBench
Agents build slices of our real projects from an empty workspace; a rubric grades them. Scores are out of 100, after intervention penalties. Every run is listed, failures included. 108 runs.
Leaderboardscored v2
Out of 100 on the Kalshi data collection, MathForge server core, Recipe Kroger cart slices, items K01–K40, M01–M30, R01–R30; scored runs, rubric v2
epoch bc153d1e05b8-cfd29c413b22-7c295401355c-aba3dbce9d85
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| opus-5-5 | 9 | 91.2 | 90.2 | 73.2–100.0 | $0.00 list $2.35 |
| gpt-6-sol | 9 | 80.9 | 92.3 | 44.8–97.1 | $0.00 list $0.41 |
| deepseek-v4-flash-0731 | 9 | 62.8 | 90.2 | 4.2–95.1 | $0.11 |
| mimo-v2.6-flash | 3 | – | 0.0 | 0.0–0.0 | $0.08 |
| mimo-v2.6-pro | 6 (3 unscored) | – | 73.8 | 68.7–76.3 | $0.18 |
Score = each slice's median run, weighted by its items (K 40, M 30, R 30).
Score per slice
All runs (36)
| date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 2026-09-28 10:20 | mimo-v2.6-pro | K | 2 | 68.7 | 28/11/1/40 | |
| 2026-09-28 09:17 | mimo-v2.6-pro | K | 1 | 73.8 | 30/9/1/40 | timeout |
| 2026-09-28 08:12 | mimo-v2.6-pro | K | 0 | 76.3 | 31/8/1/40 | timeout |
| 2026-09-28 06:45 | mimo-v2.6-pro | K | 1 | – | 0/40/0/40 | harness error: provider rate limit: Provider rate limit reached. Please retry shortly. (request_id=req_28544fd59f16) |
| 2026-09-28 06:41 | mimo-v2.6-pro | K | 0 | – | 0/40/0/40 | harness error: provider rate limit: Provider rate limit reached. Please retry shortly. (request_id=req_86de6842cc8d) |
| 2026-09-28 06:39 | mimo-v2.6-pro | K | 2 | – | 0/40/0/40 | harness error: provider rate limit: Provider rate limit reached. Please retry shortly. (request_id=req_dc3769037d1e) |
| 2026-09-28 03:30 | mimo-v2.6-flash | K | 0 | 0.0 | 0/40/0/40 | timeout |
| 2026-09-28 03:30 | mimo-v2.6-flash | K | 1 | 0.0 | 0/40/0/40 | timeout |
| 2026-09-28 03:30 | mimo-v2.6-flash | K | 2 | 0.0 | 0/40/0/40 | timeout |
| 2026-09-27 17:50 | opus-5-5 | R | 2 | 100.0 | 30/0/0/30 | |
| 2026-09-27 17:50 | opus-5-5 | R | 1 | 95.1 | 29/1/0/30 | |
| 2026-09-27 17:49 | opus-5-5 | R | 0 | 90.2 | 28/2/0/30 | |
| 2026-09-27 17:49 | gpt-6-sol | R | 2 | 97.1 | 29/1/0/30 | |
| 2026-09-27 17:49 | gpt-6-sol | R | 1 | 92.3 | 28/2/0/30 | |
| 2026-09-27 17:48 | gpt-6-sol | R | 0 | 89.4 | 27/3/0/30 | |
| 2026-09-27 17:48 | deepseek-v4-flash-0731 | R | 2 | 90.2 | 28/2/0/30 | |
| 2026-09-27 17:48 | deepseek-v4-flash-0731 | R | 1 | 88.5 | 26/4/0/30 | |
| 2026-09-27 17:47 | deepseek-v4-flash-0731 | R | 0 | 91.4 | 27/3/0/30 | |
| 2026-09-27 17:47 | opus-5-5 | K | 2 | 81.5 | 34/6/0/40 | |
| 2026-09-27 17:42 | opus-5-5 | K | 1 | 81.5 | 34/6/0/40 | |
| 2026-09-27 17:36 | opus-5-5 | K | 0 | 73.2 | 30/9/1/40 | |
| 2026-09-27 17:32 | gpt-6-sol | K | 2 | 44.8 | 20/20/0/40 | |
| 2026-09-27 17:27 | gpt-6-sol | K | 1 | 61.8 | 26/14/0/40 | |
| 2026-09-27 17:22 | gpt-6-sol | K | 0 | 63.9 | 27/13/0/40 | |
| 2026-09-27 17:17 | gpt-6-sol | M | 1 | 95.1 | 29/1/0/30 | |
| 2026-09-27 17:16 | gpt-6-sol | M | 2 | 95.1 | 29/1/0/30 | |
| 2026-09-27 17:16 | deepseek-v4-flash-0731 | K | 2 | 4.2 | 2/38/0/40 | |
| 2026-09-27 17:11 | gpt-6-sol | M | 0 | 95.1 | 29/1/0/30 | |
| 2026-09-27 17:10 | deepseek-v4-flash-0731 | K | 1 | 18.2 | 8/32/0/40 | |
| 2026-09-27 17:05 | deepseek-v4-flash-0731 | M | 2 | 95.1 | 29/1/0/30 | |
| 2026-09-27 17:05 | opus-5-5 | M | 2 | 100.0 | 30/0/0/30 | |
| 2026-09-27 17:04 | opus-5-5 | M | 1 | 89.4 | 27/3/0/30 | |
| 2026-09-27 17:03 | deepseek-v4-flash-0731 | K | 0 | 20.3 | 9/31/0/40 | |
| 2026-09-27 16:51 | opus-5-5 | M | 0 | 100.0 | 30/0/0/30 | |
| 2026-09-27 16:46 | deepseek-v4-flash-0731 | M | 0 | 94.8 | 28/1/1/30 | |
| 2026-09-27 16:40 | deepseek-v4-flash-0731 | M | 1 | 94.6 | 28/1/1/30 |
Leaderboardscored v1
Out of 100 on the Kalshi data collection, MathForge server core, Recipe Kroger cart slices, items K01–K40, M01–M30, R01–R30; scored runs, rubric v1
epoch c4dbfc7e5c5a-644f71243d5d-c182c64574b7-88e7f43dcd6d
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| opus-5-5 | 9 | 49.9 | 58.4 | 22.9–66.7 | $0.00 list $2.35 |
| gpt-6-sol | 9 | 28.2 | 22.9 | 8.4–58.1 | $0.00 list $0.43 |
| deepseek-v4-flash-0731 | 6 | – | 27.2 | 2.1–83.7 | $0.10 |
Score = each slice's median run, weighted by its items (K 40, M 30, R 30).
Score per slice
All runs (24)
| date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 2026-09-27 14:17 | deepseek-v4-flash-0731 | R | 0 | 83.7 | 25/5/0/30 | |
| 2026-09-27 14:16 | deepseek-v4-flash-0731 | R | 1 | 40.4 | 12/18/0/30 | |
| 2026-09-27 14:07 | deepseek-v4-flash-0731 | R | 2 | 42.1 | 14/16/0/30 | |
| 2026-09-27 13:36 | gpt-6-sol | R | 2 | 43.3 | 13/17/0/30 | |
| 2026-09-27 13:35 | gpt-6-sol | R | 0 | 22.9 | 8/22/0/30 | |
| 2026-09-27 13:35 | gpt-6-sol | R | 1 | 22.9 | 8/22/0/30 | |
| 2026-09-27 13:31 | opus-5-5 | R | 2 | 22.9 | 8/22/0/30 | |
| 2026-09-27 13:29 | opus-5-5 | R | 1 | 22.9 | 8/22/0/30 | |
| 2026-09-27 13:28 | opus-5-5 | R | 0 | 22.9 | 8/22/0/30 | |
| 2026-09-27 13:18 | gpt-6-sol | M | 2 | 58.1 | 18/11/1/30 | |
| 2026-09-27 13:17 | gpt-6-sol | M | 0 | 54.8 | 17/12/1/30 | |
| 2026-09-27 13:17 | gpt-6-sol | M | 1 | 55.3 | 17/12/1/30 | |
| 2026-09-27 13:11 | opus-5-5 | M | 0 | 61.0 | 19/10/1/30 | |
| 2026-09-27 13:11 | opus-5-5 | M | 2 | 58.4 | 19/11/0/30 | |
| 2026-09-27 13:10 | opus-5-5 | M | 1 | 55.3 | 17/12/1/30 | |
| 2026-09-27 12:59 | gpt-6-sol | K | 2 | 14.0 | 6/34/0/40 | |
| 2026-09-27 12:55 | gpt-6-sol | K | 1 | 8.4 | 4/36/0/40 | |
| 2026-09-27 12:50 | opus-5-5 | K | 2 | 66.7 | 29/10/1/40 | |
| 2026-09-27 12:45 | opus-5-5 | K | 1 | 63.7 | 29/11/0/40 | |
| 2026-09-27 12:39 | gpt-6-sol | K | 0 | 11.9 | 5/35/0/40 | |
| 2026-09-27 12:28 | opus-5-5 | K | 0 | 63.7 | 29/11/0/40 | |
| 2026-09-27 12:23 | deepseek-v4-flash-0731 | K | 2 | 2.1 | 1/39/0/40 | |
| 2026-09-27 12:10 | deepseek-v4-flash-0731 | K | 0 | 14.0 | 6/34/0/40 | |
| 2026-09-27 12:06 | deepseek-v4-flash-0731 | K | 1 | 4.2 | 2/38/0/40 |
Leaderboardcalibration
Out of 100 on the Kalshi data collection slice, items K01–K40; calibration runs, rubric not locked
epoch c4dbfc7e5c5a-644f71243d5d-c182c64574b7-88e7f43dcd6d
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| opus-5-5 · harness 2 | 3 | 75.0 | 75.0 | 60.0–75.0 | $0.00 list $3.50 |
| opus-5-5 · harness 3 | 3 | 60.0 | 60.0 | 60.0–65.0 | $0.00 list $4.61 |
| gpt-6-sol · harness 3 | 3 | 50.0 | 50.0 | 25.0–65.0 | $0.00 list $0.48 |
| gpt-6-sol · harness 2 | 3 | 25.0 | 25.0 | 25.0–30.0 | $0.00 list $0.65 |
| deepseek-v4-flash-0731 · harness 2 | 3 | 10.0 | 10.0 | 5.0–15.0 | $0.19 |
| deepseek-v4-flash-0731 · harness 3 | 3 | 5.0 | 5.0 | 5.0–5.0 | $0.21 |
| deepseek-v4-flash-0731 · IDE | 1 (1 unscored) | – | – | – | $0.00 |
| deepseek-v4.1-flash (run as deepseek-v4-flash-0731) · IDE | 1 (1 unscored) | – | – | – | unmeasured |
Score = each slice's median run, weighted by its items (K 40).
Score per slice
All runs (20)
| date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 2026-09-27 11:50 | deepseek-v4.1-flash (run as deepseek-v4-flash-0731) · IDE | K | 91 | – | 30/10/0/40 | harness error: routing: pinned proxy served nothing, expected only deepseek/deepseek-v4-flash-0731 |
| 2026-09-27 11:05 | opus-5-5 · harness 3 | K | 2 | 60.0 | 12/8/0/20 | timeout |
| 2026-09-27 10:49 | opus-5-5 · harness 3 | K | 1 | 60.0 | 12/8/0/20 | |
| 2026-09-27 10:36 | gpt-6-sol · harness 3 | K | 2 | 25.0 | 5/15/0/20 | |
| 2026-09-27 10:27 | gpt-6-sol · harness 3 | K | 1 | 65.0 | 13/7/0/20 | |
| 2026-09-27 10:16 | gpt-6-sol · harness 3 | K | 0 | 50.0 | 10/10/0/20 | |
| 2026-09-27 10:05 | deepseek-v4-flash-0731 · harness 3 | K | 0 | 5.0 | 1/19/0/20 | |
| 2026-09-27 10:01 | opus-5-5 · harness 3 | K | 0 | 65.0 | 13/7/0/20 | |
| 2026-09-27 09:55 | deepseek-v4-flash-0731 · harness 3 | K | 2 | 5.0 | 1/19/0/20 | |
| 2026-09-27 09:43 | deepseek-v4-flash-0731 · harness 3 | K | 1 | 5.0 | 1/19/0/20 | |
| 2026-09-27 09:18 | deepseek-v4-flash-0731 · IDE | K | 90 | – | 27/13/0/40 | harness error: routing unverified (IDE run before the proxy fix); 3 intervention(s), −9 |
| 2026-09-27 07:21 | opus-5-5 · harness 2 | K | 2 | 75.0 | 15/5/0/20 | |
| 2026-09-27 06:56 | gpt-6-sol · harness 2 | K | 2 | 30.0 | 6/14/0/20 | |
| 2026-09-27 06:45 | gpt-6-sol · harness 2 | K | 1 | 25.0 | 5/15/0/20 | |
| 2026-09-27 06:39 | gpt-6-sol · harness 2 | K | 0 | 25.0 | 5/15/0/20 | |
| 2026-09-27 06:31 | opus-5-5 · harness 2 | K | 1 | 60.0 | 12/8/0/20 | |
| 2026-09-27 06:26 | deepseek-v4-flash-0731 · harness 2 | K | 1 | 15.0 | 3/17/0/20 | |
| 2026-09-27 06:18 | opus-5-5 · harness 2 | K | 0 | 75.0 | 15/5/0/20 | |
| 2026-09-27 06:04 | deepseek-v4-flash-0731 · harness 2 | K | 0 | 10.0 | 2/18/0/20 | |
| 2026-09-27 05:59 | deepseek-v4-flash-0731 · harness 2 | K | 2 | 5.0 | 1/19/0/20 |
Leaderboardcalibration
Out of 100 on the Kalshi data collection slice, items K01–K20; calibration runs, rubric not locked
epoch 46fc803f1a9c-ec54f73daa8b-c182c64574b7
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | 3 | 60.0 | 60.0 | 50.0–65.0 | $0.35 |
| opus-5-5 | 4 (2 unscored) | 60.0 | 60.0 | 60.0–60.0 | $0.00 list $1.39 |
| gpt-6-sol | 3 | 45.0 | 45.0 | 25.0–45.0 | $0.00 list $0.45 |
Score = each slice's median run, weighted by its items (K 20).
Score per slice
All runs (10)
| date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 2026-09-27 05:24 | opus-5-5 | K | 1 | 60.0 | 12/8/0/20 | |
| 2026-09-27 02:47 | gpt-6-sol | K | 2 | 45.0 | 9/11/0/20 | |
| 2026-09-27 02:43 | gpt-6-sol | K | 1 | 45.0 | 9/11/0/20 | |
| 2026-09-27 02:42 | deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | K | 1 | 65.0 | 13/7/0/20 | |
| 2026-09-27 02:36 | gpt-6-sol | K | 0 | 25.0 | 5/15/0/20 | |
| 2026-09-27 02:33 | deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | K | 2 | 60.0 | 12/8/0/20 | |
| 2026-09-27 02:28 | opus-5-5 | K | 2 | – | 0/20/0/20 | harness error: agent never started (exit 1, 0 tokens) |
| 2026-09-27 02:28 | opus-5-5 | K | 1 | – | 0/20/0/20 | harness error: agent never started (exit 1, 0 tokens) |
| 2026-09-27 02:28 | opus-5-5 | K | 0 | 60.0 | 12/8/0/20 | |
| 2026-09-27 02:10 | deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | K | 0 | 50.0 | 10/10/0/20 |
Leaderboardcalibration
Out of 100 on the Kalshi data collection slice, items K01–K20; calibration runs, rubric not locked
epoch fe823afec90b-7c33790e499b-c182c64574b7
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | 3 | 65.0 | 65.0 | 45.0–65.0 | $0.34 |
| gpt-6-sol | 3 | 45.0 | 45.0 | 40.0–55.0 | $0.00 list $0.93 |
| opus-5-5 | 3 (3 unscored) | – | – | – | $0.00 |
Score = each slice's median run, weighted by its items (K 20).
Score per slice
All runs (9)
| date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 2026-09-27 01:44 | gpt-6-sol | K | 2 | 55.0 | 11/9/0/20 | |
| 2026-09-27 01:37 | gpt-6-sol | K | 1 | 45.0 | 9/11/0/20 | |
| 2026-09-27 01:34 | deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | K | 0 | 45.0 | 9/11/0/20 | |
| 2026-09-27 01:28 | gpt-6-sol | K | 0 | 40.0 | 8/12/0/20 | |
| 2026-09-27 01:19 | deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | K | 2 | 65.0 | 13/7/0/20 | |
| 2026-09-27 01:16 | deepseek-v4.1-flash (run as deepseek-v4-flash-0731) | K | 1 | 65.0 | 13/7/0/20 | |
| 2026-09-27 01:16 | opus-5-5 | K | 0 | – | 0/20/0/20 | harness error: agent never started (exit 1, 0 tokens) |
| 2026-09-27 01:16 | opus-5-5 | K | 1 | – | 0/20/0/20 | harness error: agent never started (exit 1, 0 tokens) |
| 2026-09-27 01:16 | opus-5-5 | K | 2 | – | 0/20/0/20 | harness error: agent never started (exit 1, 0 tokens) |
Leaderboardunknown
No scores recorded for these runs (result.json missing or unreadable)
epoch unknown
| model | runs | score | median run | range | $/run |
|---|---|---|---|---|---|
| deepseek-v4-flash-0731 | 3 (3 unscored) | – | – | – | $0.00 |
| mimo-v2.6-pro | 5 (5 unscored) | – | – | – | $0.00 |
| opus-5-5 | 1 (1 unscored) | – | – | – | $0.00 |
Score per slice
No scored runs to chart.
All runs (9)
| date (UTC) | model | slice | seed | score | pass/fail/part/total | notes |
|---|---|---|---|---|---|---|
| 2026-09-28 07:05 | mimo-v2.6-pro | K | 1 | – | 0/0/0/0 | no result.json |
| 2026-09-28 07:02 | mimo-v2.6-pro | K | 0 | – | 0/0/0/0 | no result.json |
| 2026-09-28 03:30 | mimo-v2.6-pro | K | 0 | – | 0/0/0/0 | no result.json |
| 2026-09-28 03:30 | mimo-v2.6-pro | K | 1 | – | 0/0/0/0 | no result.json |
| 2026-09-28 03:30 | mimo-v2.6-pro | K | 2 | – | 0/0/0/0 | no result.json |
| 2026-09-27 13:37 | deepseek-v4-flash-0731 | M | 0 | – | 0/0/0/0 | no result.json |
| 2026-09-27 13:37 | deepseek-v4-flash-0731 | M | 1 | – | 0/0/0/0 | no result.json |
| 2026-09-27 13:37 | deepseek-v4-flash-0731 | M | 2 | – | 0/0/0/0 | no result.json |
| 2026-09-27 04:54 | opus-5-5 | K | 2 | – | 0/0/0/0 | no result.json |