Bug Hunt Bench — which AI coding model fixes the most real bugs?

105 real bugs, hidden in two production codebases. Frontier coding models — GPT-6, Claude, Grok, Gemini, DeepSeek, Kimi, GLM and more — get one round per repo in their own agentic CLI (Codex CLI, Claude Code, Grok CLI, Antigravity CLI) to find and fix what they can. Every diff is graded blind against a withheld answer key. The score counts planted bugs only.

Live board: bughunt.productcompass.pm · Method, caveats and definitions · Raw data · Findings by wave

If the numbers save you a benchmark run of your own, star this repo — that is what keeps the bench findable, and new models are added as they ship.

Updated Oct 14, 2026 · 185 rows from 227 scored runs · 46 models · 26 of 105 bugs have never been fixed by any model.

Current leader: Sonnet 5.5 at max effort — 51.3 / 105 (22.7/45 on repo 1, 28.7/60 on repo 2).

Best per lab: Anthropic: Sonnet 5.5 (max) 51.3 · OpenAI: GPT-6 Astra (max) 45 · Meta: Muse Wonder Full 1.0 (max) 42 · Google: Gemini 4 Argon (high) 40.7 · Unbiased: Pareto (ex-Union Alpha) (default) 30.7 · xAI: Grok 4.7 (xhigh) 29 · Alibaba: Qwen3.8-Flash (max) 26 · Xiaomi: MiMo-V2.6-Flash (default) 23.3 · DeepSeek: DeepSeek V4.1 Flash (max) 21.7 · Moonshot AI: Kimi K3 (default) 21 · Z.ai: GLM-5.3 (max) 19 · Tencent: Hy4 Preview (default) 18 · Mistral: Mistral Large 4 (high) 15

#	Model	Harness	Effort	Runs	Fixed /105	Repo 1 /45	Repo 2 /60	Extras	Wall	Cost	Date
1	Sonnet 5.5	Claude Code	max	3	51.3	22.7	28.7	26.3	287 min	$154.03	2026-09-29
2	GPT-6 Astra	Codex CLI	max	3	45	22	23	55	90 min	$33.03	2026-09-14
3	GPT-6.1 Sol	Codex CLI	max	3	44.3	22.3	22	65.3	139 min	$5.88	2026-09-30
4	GPT-5.6 Sol	Codex CLI	max	2	43.5	19	24.5	48.5	253 min	$95.35	2026-09-16
5	GPT-6 Astra	Codex CLI	xhigh	1	43	23	20	53	59 min	$24.22	2026-09-04
6	Fable 5.1	Claude Code	max	1	43	19	24	11	73 min	$87.18	2026-09-01
7	GPT-6.1 Sol	Codex CLI	xhigh	3	42.7	22.3	20.3	56.7	99 min	$4.35	2026-09-30
8	Muse Wonder Full 1.0	Muse Code / Meta API	max	2	42	21.5	20.5	36.5	145 min	—	2026-10-14
9	Opus 5.5	Claude Code	max	3	41.7	19	22.7	9	67 min	$58.53	2026-09-23
10	Muse Wonder Full 1.0	Muse Code / Meta API	xhigh	2	41.5	16	25.5	24.5	90 min	—	2026-10-14
11	Gemini 4 Argon	Antigravity CLI	high	3	40.7	13	27.7	7.3	56 min	$7.16	2026-10-14
12	Gemini 4 Argon	Antigravity CLI	medium	3	39.3	13.7	25.7	11.3	60 min	$7.50	2026-10-14
13	GPT-5.6 Sol	Codex CLI	xhigh	1	39	18	21	59	126 min	$52.75	2026-08-28
14	GPT-6.1 Sol	Codex CLI	high	2	36.5	17	19.5	40	70 min	$3.26	2026-09-30
15	Opus 5.5	Claude Code	xhigh	3	36	17	19	13.3	42 min	$34.98	2026-09-23
16	Sonnet 5.5	Claude Code	xhigh	3	36	16	20	14	73 min	$42.86	2026-09-29
17	Gemini 4 Argon	Antigravity CLI	low	3	35.3	12.7	22.7	5.7	60 min	$6.47	2026-