AIxCC: the closest thing to a proof

DARPA's AI Cyber Challenge ran 90+ teams down to seven finalists across two years, $29.5M in prizes, and 54+ million lines of real open-source code, and produced the only public, third-party-witnessed dataset of what autonomous vulnerability-finding-and-patching actually costs and where it breaks. The headline numbers are good: 86% of planted bugs found, 68% patched. The buried number is the one that matters for a business plan: every real-world C vulnerability found in the finals went unpatched. Zero out of six.

#Structure, prize money, timeline

AIxCC was a DARPA program, run jointly with ARPA-H, asking teams to build fully autonomous Cyber Reasoning Systems (CRSs) — find and patch vulnerabilities in real open-source codebases with no human in the loop. It was explicitly pitched as the LLM-era successor to DARPA's 2016 Cyber Grand Challenge. Anthropic, Google, Microsoft, and OpenAI were named collaborators, contributing model credits rather than prize money (darpa.mil/research/programs/ai-cyber).

• August 2023 — Program launched by the White House and DARPA with an initial $20M+ prize pool (whitehouse.gov).
• December 2023 — Registration opens, $4M added to the pool, scoring algorithm published.
• March 2024 — ARPA-H co-sponsors; small-business track selects participants for $1M each in development funding.
• August 2024, DEF CON 32 (Semifinal / ASC) — ~90 teams registered, 42 submitted CRSs. Seven advanced, each awarded $2M.
• August 2025, DEF CON 33 (Final / AFC) — Seven finalists compete for an $8.5M pool ($4M / $3M / $1.5M for 1st–3rd, non-monetary recognition for 4th–7th).
• Cumulative program prize pool: $29.5M, including the small-business track (darpa.mil/news/2024/ai-cyber-challenge-cybersecurity, darpa.mil/news/2025/ai-cyber-challenge-scoring).

Scoring: finding a vulnerability meant submitting a Proof of Vulnerability (PoV) that crashed a provided fuzzing harness. Patching was worth roughly 3x as many points as finding — and a patch had to survive every crashing input any competing team found for that bug, not just the discoverer's own, which rewarded genuinely correct fixes over narrow ones. Two bug categories existed: organizer-planted synthetic vulnerabilities, and real-world vulnerabilities already latent in the challenge codebases, which were responsibly disclosed to maintainers when found. Challenge projects spanned the Linux kernel, nginx, SQLite3, Apache Tika, Jenkins, and an expanded final-round set of 28+ C and Java repos (aicyberchallenge.com/final-competition-procedures-and-scoring-guide).

Every finalist was contractually required to open-source its CRS under an OSI-approved license to keep its prize money. All seven did.

#Final standings, with prizes

Rank	Team	Prize	Score	Vulns found	Patches	Total spend	Cost/point
1	Team Atlanta (ATLANTIS)	$4,000,000	392.76	43	41 of 47 produced (87.2%)	$103,300 ($73.9K compute + $29.4K LLM) [unverified]	$263 [unverified]
2	Trail of Bits (Buttercup)	$3,000,000	219	28	19, across 48 challenges / 23 repos	$39,600 ($21.1K LLM + $18.5K compute) [unverified]	$181 [unverified]
3	Theori (RoboDuck)	$1,500,000	211	34	20	$31,800 [unverified]	$151 [unverified]
4	all_you_need_is_a_fuzzing_brain	—	154	28	14	not disclosed	—
5	Shellphish (ARTIPHISHELL)	—	136	28	11	not disclosed	—
6	42-b3yond-6ug	—	105	41	3	not disclosed	—
7	Lacrosse (SIFT)	—	10	1	1	not disclosed	—

Sources: aicyberchallenge.com/def-con-33, blog.trailofbits.com/2025/08/09, Team Atlanta's ATLANTIS paper (arXiv:2509.14589). The per-team spend and cost-per-point figures come from Trail of Bits' own comparative post-competition analysis, not from DARPA, and are marked [unverified] accordingly.

The rank-order anomaly tells the real story: 42-b3yond-6ug found 41 vulnerabilities — more than Trail of Bits (28) or Theori (34) — but patched only 3, and finished 6th. Team Atlanta found roughly the same number (43) and patched 41, and won outright. Discovery was not the differentiator by 2025; conversion to a verified, correct patch was.

#The seven CRS architectures

Team / system	Architecture	LLM vs. fuzzing split	Models used	Repo	License	Maintenance
Team Atlanta / ATLANTIS (1st)	Two-tier K8s (CRS-level shared services + CP-level per-challenge work); language-specialized modules (-C, -Java, -Multilang, -Patch, -SARIF); CodeQL + SymCC + SVF; ensemble fuzzing (LibAFL/AFL++/libFuzzer with automatic fallback); custom fuzzer BULLSEYE beat AFL++ on 7/11 targets.	Balanced — LLMs generate harnesses and drive patch agents, but every output is treated as a hypothesis requiring static/dynamic verification.	Claude (Opus 4, Sonnet 4/3.7/3.5, Haiku), GPT-4o/o3/o4-mini, Gemini, plus a custom GRPO-fine-tuned patch model on Azure.	aixcc-afc-atlantis (615★)	MIT	Archived; sibling repos (crs-claude-code) actively developed in 2026.
Trail of Bits / Buttercup (2nd)	Modular services (Orchestrator, Seed Generator, Fuzzer w/ libFuzzer+Jazzer, Patcher, Scheduler), Redis-backed, K8s/Helm/Terraform on Azure; 300–500 parallel fuzzing agents seeded by LLM-generated inputs.	Fuzzing-anchored; 100,000+ LLM requests in the final, highest raw call volume of any team, via cheaper models at high volume.	Claude Sonnet 4, GPT-4.1, GPT-4.1-mini, via a provider-agnostic LiteLLM abstraction.	buttercup (1.6k★)	AGPL-3.0	Active, v1.0 shipped Aug 2025, 884 commits.
Theori / RoboDuck (3rd)	Multi-agent, Docker/Terraform/Azure; static analysis (Infer) filters false positives before LLM handoff.	Most AI-forward of the top three — team lead said it "would still function" with non-AI components stripped. Aggressive 2:1 speculative-patch-to-PoV ratio.	Claude Sonnet 4, o3/o4-mini, Gemini, Azure-hosted models via config.	aixcc-afc-archive (269★)	AGPL-3.0	Archived Feb 2026; README: "will NOT be supported."
all_you_need_is_a_fuzzing_brain (4th)	Pipeline: target analysis → fuzzer building → direction planning → Suspicious-Point agents → PoV construction; integrates OSS-Fuzz toolchain.	Most AI-centric of all seven — ~90% of PoVs from direct LLM reasoning, fuzzing as fallback/validation only. Every finding must trigger an actual crash before reporting.	Multi-provider (Anthropic, OpenAI, Google) via CLI/REST/MCP.	afc-crs-...fuzzing-brain (134★)	Apache-2.0	Active, 290 commits.
Shellphish / ARTIPHISHELL (5th)	Fuzzing/static-analysis anchored, plus "Grammar Guy" — an LLM generating and evolving input grammars from live coverage feedback. Never submitted an unverified patch.	LLM-augmented fuzzing, not LLM-primary.	Not disclosed.	artiphishell (138★)	MIT	Academic team, ongoing research use.
42-b3yond-6ug / BugBuster (6th)	Terraform + K8s; "super patches" — one patch engineered to fix multiple unrelated bugs at once, trading precision for coverage.	Not detailed publicly.	Not disclosed.	42-b3yond-6ug-crs (30★)	GPL-3.0	Single-commit snapshot.
Lacrosse (SIFT) (7th)	Multi-host Azure, Terraform, Tailscale, orchestrator "Optimus Zero," LLM-consensus across providers.	Multi-provider consensus-checking.	Gemini, Claude, OpenAI.	afc-crs-lacrosse (9★)	MIT	No public retrospective; weakest result despite comparable sophistication.

Architecture sophistication did not predict outcome — Lacrosse's stack looks comparable to mid-table teams on paper and scored 10 points. See How these systems are actually built.

#The hard numbers, corrected

DARPA's own materials circulated two different figures after the finals — 63 and 70 as the total planted synthetic-vulnerability count, producing two detection-rate readings (86% vs. 77%) and two patch-rate readings (68% vs. 61%). Verification against DARPA's official finals page resolves this: 63 total synthetic vulnerabilities, 86% found (54 unique), 68% patched (43) (aicyberchallenge.com/finals-winners-announcement). The 77%/61% figures should not be repeated as correct.

The figure that matters more than any leaderboard position: of the vulnerabilities found in the real, non-synthetic challenge codebases during the final, sources put the count at 18 real-world zero-days — 6 in C, 12 in Java — with 11 patched, all on the Java side. Zero of the six real C vulnerabilities were successfully patched by any team. Both the 18-count and the ~$152 average cost-per-task figure could not be independently confirmed against DARPA's own site content and are marked [unverified] — but the shape of the C-versus-Java split is corroborated by team-level retrospectives and is the single most load-bearing fact in this dataset regardless of the exact zero-day count.

Cost, marked [unverified] where the source is press coverage rather than DARPA's own site: the ~$152/completed-task average and per-team totals ($103,300 Team Atlanta, $39,600 Trail of Bits, $31,800 Theori) come from Trail of Bits' own cost analysis and secondary press, not an independently reconciled DARPA figure. Trail of Bits' central claim — 2nd place at roughly a third of Team Atlanta's spend while making comparable or higher LLM call volume — is corroborated by their own retrospective, but should be read as a self-reported efficiency claim, not an audited number.

#What the teams said was hard

• Dan Guido (Trail of Bits CEO): "There's just too much code to look through, and it's too complex to process in order to find all the vulnerabilities" — exhaustive LLM-driven review doesn't scale even at frontier quality. He also said commercial LLM safety guardrails actively interfered with reasoning usefully about exploitable vulnerabilities, a friction point rarely discussed outside this competition (axios.com).
• The autonomy-vs-safety tension shaped every architecture. Teams needed enough autonomy to run unattended while avoiding shipping a bad patch to critical infrastructure — itself a security incident. Team Atlanta and Shellphish both disabled unverified-patch submission before the final.
• Fuzzer fragility ate real engineering time. Team Atlanta's LibAFL suffered instrumentation crashes under sustained load, forcing a cascading fallback to AFL++ and libFuzzer.
• Static analysis doesn't scale uniformly. CodeQL took "several hours" on SQLite3 and produced false negatives on curl, per ATLANTIS's own paper, forcing a hybrid pipeline over trust in any single tool.
• The "last mile" in Java exploitation: locating a vulnerable sink isn't the same as triggering it. Team Atlanta built a dedicated component (ExpKit) because LLM agents could find vulnerable paths but routinely failed to construct the object graphs needed to trigger the bug.
• Verification-gated LLM output was universal, not optional. Every team that discussed it converged on "never trust an LLM claim without triggering an actual sanitizer crash" — LLM hallucination of both vulnerabilities and fixes is a designed-around failure mode across all seven systems.

#After the finals: Patch the Planet and the human-in-the-loop reality

DARPA's own transition plan has two tracks: DARPA itself applying the open-sourced CRSs to real critical-infrastructure repositories, and teams commercializing the technology. An additional pool — reported as $1.4M total or $200,000 per team — was earmarked for post-competition integration work.

The clearest real-world descendant is Trail of Bits × OpenAI's "Patch the Planet," launched June 22, 2026 (blog.trailofbits.com/2026/06/22). It pairs Trail of Bits security engineers with maintainers of real OSS projects, giving selected maintainers six months of ChatGPT Pro access including Codex security features. The corrected figures, reconciled against Trail of Bits' own posts: roughly 19–30 projects under review, 51 issues filed, 64 pull requests submitted, 37 patches merged. An earlier circulating figure of "50 repos, 1,268 issues, 175 merged patches" does not match anything Trail of Bits has published and should not be repeated.

The important structural finding: Patch the Planet is Buttercup-derived tooling plus human security engineers plus AI coding assistants — not the fully autonomous CRS that won DEF CON. That gap between "competition demonstrates autonomous CRS" and "production deployment is human-supervised AI-assisted review" is the most important adoption-reality finding in this retrospective, and should discipline any pitch that leans on AIxCC as evidence of ready-for-production autonomy. Buttercup itself remains a live, actively-developed open-source project (AGPL-3.0) independent of the collaboration — see The open-source stack.

Beyond Patch the Planet: Samsung Research is folding Team Atlanta's techniques into its own product security org (no standalone spinout identified); Theori was already a funded offensive-security company pre-AIxCC and continues its existing product line without an announced direct RoboDuck pipeline; Shellphish remains academic; no confirmed spinouts exist for 42-b3yond-6ug or Lacrosse. No direct code integration between any AIxCC CRS and OSS-Fuzz was found — they remain parallel efforts.

#Comparable industry results

System	Org	Method	Headline result	Cost / notes
AIxCC final (aggregate)	DARPA / 7 finalist teams	LLM + fuzzing + static analysis, competition-scoped	86% found / 68% patched of 63 planted vulns; 18 real zero-days found, 11 patched (all Java) [unverified count]	~$152/task [unverified]; $151–$263/point among top 3 [unverified]
OSS-Fuzz-Gen	Google	LLM-generated fuzz harnesses fed into existing OSS-Fuzz infra, not a full vuln-finding agent	30 new bugs found that were unreachable by prior human-written harnesses; large coverage gains across 297 projects	Harness-generation cost only, cheaper than an autonomous-agent approach since it augments existing infra (github.com/google/oss-fuzz-gen)
Google Big Sleep	Project Zero + DeepMind	Variant analysis — given a previously-fixed bug's diff, finds similar unpatched instances elsewhere	First public AI-agent discovery of a previously-unknown, exploitable memory-safety bug in production software (SQLite seriesBestIndex stack buffer underflow, Oct 2024); 150 CPU-hours of AFL fuzzing had failed to find the same bug	Single documented case, not a throughput statistic (projectzero.google)
OpenAI Aardvark ("Codex Security")	OpenAI	Agentic threat-modeling + commit scanning + sandbox validation + Codex-generated patch proposal for human review	92% detection on a curated benchmark of known + synthetic vulnerabilities; 10 CVEs assigned from real OSS findings	At research preview, not GA, as of Mar 2026; no per-scan cost disclosed (openai.com/index/introducing-aardvark)

Reading these side by side requires care: Aardvark's 92% is detection on a curated benchmark with a known answer key, while AIxCC's 86% is detection of genuinely novel planted bugs under time pressure — not directly comparable methodologies, even though both get quoted as "detection rate." Big Sleep's single SQLite find is a proof-of-concept case study, not a rate — it establishes only that variant analysis can outperform fuzzing on the right bug class, not a throughput number. See What the frontier labs do themselves for how Big Sleep and Aardvark fit into each lab's broader safety and product posture.

#What AIxCC proves, and what it implies about cost-per-vulnerability economics

Proves possible: end-to-end autonomous find-and-patch pipelines work on real, large (54M+ LOC) codebases at genuinely low unit cost and fast turnaround, when scoped to languages and bug classes the tooling is tuned for. LLM-plus-program-analysis clearly beats either alone — every top-3 team combined fuzzing/symbolic execution/static analysis with LLM agents, and none of the top finishers succeeded pure-LLM or pure-fuzzing. Real zero-days are a reachable side effect of general-purpose CRS work, not just a synthetic-benchmark exercise.

Proves still hard: patch reliability for C and native-memory bugs lags discovery badly — the 6-found/0-patched split on real C zero-days is the sharpest data point available anywhere on this gap. Discovery volume and patch-conversion rate are not correlated (42-b3yond-6ug: 41 found/3 patched vs. Team Atlanta: 43 found/41 patched), meaning a product that reports "N vulnerabilities found" without a credible fix-conversion story is measuring the wrong half of the problem for a buyer who wants remediated risk, not a longer backlog. Operational engineering — fuzzer stability, static-analysis scaling, multi-VM orchestration — ate a large, under-reported share of every team's effort, and commercial LLM safety guardrails can directly impede vulnerability-hunting workloads, a real and documented obstacle rather than a hypothetical one.

Cost-per-vulnerability economics, stated carefully: achievable unit economics in a highly-optimized, competition-tuned setting fell in a wide range — roughly $150–$260 per point-scoring vulnerability action in the top-three teams' aggregate figures [unverified], against Trail of Bits' claim of comparable results at roughly a third of the top team's spend. LLM inference and compute/fuzzing infrastructure were roughly comparable in magnitude as cost drivers, which argues against a mental model where "it's all just API calls" — fuzzing infrastructure remains a first-order cost, directly relevant to Unit economics and the compute bill for anyone modeling a defensive-code product's margins. The real deployment evidence — Patch the Planet's roughly 51 issues to 37 merged patches across ~19–30 real projects with human oversight — suggests something like a 60–70% issue-to-merge conversion rate is realistic at production scale, a far more conservative planning number than the 87–95% patch-success rates the top competition teams hit inside a tightly-scoped, time-boxed contest. Anyone pricing a defensive-code product against AIxCC's competition numbers is pricing against the wrong regime.

#What this means for us

• The C/Java patch asymmetry is the single most decision-relevant fact here: if the product plan touches C/C++ memory-safety remediation, plan for discovery-only tooling as the near-term ceiling, not autonomous patching. Java and other memory-safe codebases are a materially easier wedge.
• Discovery-without-conversion is a trap replicated across three data points (42-b3yond-6ug, AIxCC's C-bug patch rate, and the field's discovery-heavy benchmark culture in Cyber benchmarks and evals). Any pitch built on "we find N vulnerabilities" needs a credible conversion story or it's selling half a product.
• Patch the Planet's human-in-the-loop model, not Buttercup's competition-mode full autonomy, is the closer analog to a defensible 2026 go-to-market — see Go to market and AI code security companies.
• All seven CRS codebases are open-source (MIT/Apache-2.0/AGPL-3.0/GPL-3.0) and mostly archived, not maintained — treat them as architecture references, not something to fork and ship. See The open-source stack.
• Every figure sourced from press rather than DARPA's own site here is marked [unverified] for a reason — coverage after the finals contained real, reconciled inconsistencies (63 vs. 70 vulnerabilities; $150K vs. $350K in AI credits). Don't repeat AIxCC numbers externally without checking which bucket they're in; see Verification ledger.
• Guido's guardrail-interference point is a real product-design constraint: a defensive tool built on off-the-shelf hosted models needs a strategy — dedicated API tiers, a provider relationship, or targeted post-training — for when safety tuning blocks legitimate vulnerability research. See Post-training playbook and Data, and whether a moat is possible.