Cyber benchmarks and evals

Offense has six mature, cross-lab-cited benchmarks and five internal frontier-lab suites; defense has one. That asymmetry is real, but it is not an opening for a two-person lab — closing it requires a continuously-refreshed real-incident pipeline and government co-validation, neither of which a startup can bootstrap in year one. Building a benchmark is a marketing move, not a business. Treat it that way.

#The master table

Twenty-plus named benchmarks exist as of August 2026; most are irrelevant to a commercial decision. The table below is the set that actually shows up in frontier-lab system cards, academic citation, or has a maintained public leaderboard.

Benchmark	Author	Year	Task format	Task count	SOTA (as of Aug 2026)	Saturated?	Cited by frontier labs?	License	Harness usable?
Cybench	Stanford	2024	CTF w/ partial-credit subtasks	40	~93–100% on tracked subsets, Claude Opus 4.6/4.7/"Mythos Preview" (Jan–Jun 2026) ai2027-tracker.com	Easy/medium tier yes; hard tier no	Yes — Anthropic RSP	CC-BY-4.0	Yes, most-reused harness in the space
NYU CTF Bench	NYU	2024	CTF (CSAW-derived)	~200	GPT-5.3-Codex 77.6%, self-reported, 3 models tracked llm-stats.com	Leaderboard thin and stale	No	Open (repo-level)	Moderate
InterCode-CTF	Princeton NLP	2023	Interactive shell CTF	~100 (picoCTF)	N/A — retired as headline eval	Yes, effectively retired	No	MIT-ish	Baseline only
3CB	Apart Research	2024	Original (non-CTF) tasks	Extensible, unspecified	No public leaderboard	Unknown	No	Open	Framework, low adoption
CyberSecEval 1–4	Meta	2023–2025	Static insecure-code gen	Instruction + autocomplete tracks	MAI-Thinking-1, 0.630 — only model tracked llm-stats.com	Public leaderboard nearly dormant	No	MIT	Yes, but low public uptake
CyberSOCEval	Meta + CrowdStrike	2025	Malware analysis + CTI reasoning	Unspecified	Not yet independently tracked	New	Not yet	CC BY 4.0	Yes, new
CyberGym	UC Berkeley / Berkeley RDI	2025	PoC reproduction from real vulns	1,507 vulns / 188 projects	Claude Opus 4.5, 50.63% pass@1 across 1,505 tasks (Nov 2025)	No — large model-over-model gains	Yes — Anthropic RSP	Open	Yes, used as a live RSP eval
CVE-Bench (exploit)	UIUC	2025	Exploit real CVEs, sandboxed web apps	40 CVEs	~13% at launch; OpenAI runs it internally on GPT-5.5	No	Yes — OpenAI	Apache-2.0	Yes, actively maintained
CVE-Bench (repair)	NAACL 2025	Patch real CVEs	509 vulns / 120 repos	21% best (SWE-agent style, 2025)	No	No	CC BY 4.0	Moderate
BountyBench	Stanford	2025	Detect / Exploit / Patch, dollar-scored	25 systems, 40 real bounties	Split by lab: OpenAI wins patching, Anthropic wins exploitation	No	Not yet	CC BY 4.0	New, promising
AIRTBench	Dreadnode	2025	Black-box AI/ML CTF	70 challenges	Claude 3.7 Sonnet, 43/70	No	No	CC BY-SA 4.0	Yes
SecRepoBench	Academic	2025	Secure code completion, real repos	318 tasks / 27 repos	Agents beat standalone LLMs	No	No	CC BY 4.0	Yes
AutoPenBench	Academic	2024	Multi-host pentest, milestones	33 tasks	21% autonomous / 64% semi-autonomous (GPT-4o/o1)	No	No	Unspecified	Yes
Frontier-lab internal suites (ExploitBench, OSS-Fuzz, Firefox 147, CyScenarioBench, ExploitGym — Anthropic; CVE-Bench, Cyber Range, VulnLMP — OpenAI; Key Skills Benchmark — DeepMind)	Anthropic / OpenAI / DeepMind	2025–2026	Varies, RSP/Preparedness/FSF-gated	41–920 items	See the What the frontier labs do themselves page	No — used to show generation-over-generation jumps	By definition	Not public	Not independently runnable
CyberBattleSim / CybORG / CAGE 1–4	Microsoft / TTCP	2021–2026	Multi-agent RL, network graph	Varies	N/A — RL policies, not an LLM leaderboard	N/A	No	Open (MIT-ish)	Yes, mature

#Offensive CTF benchmarks, and why they saturate

Cybench, NYU CTF Bench, InterCode-CTF, and 3CB are the four public CTF-style suites, and three of the four are effectively dead as headline evals. InterCode-CTF is now cited only as a baseline sanity check — one paper reruns it to show a plain ReAct agent with no special scaffolding already solves most of it. NYU CTF Bench's leaderboard tracks exactly three models and flags its own numbers as self-reported. 3CB never shipped a leaderboard at all.

Cybench is the survivor, and it survives on borrowed time. Its easy and medium tiers are saturated: Claude Opus 4.6 hit 93% on a 37-problem subset in January 2026, Opus 4.7 hit 96%, and a preview build reportedly reached 100% on a 35-problem subset by June 2026 — undisclosed subset scores from system cards, not the original 40-task suite, which makes cross-model comparison noisier than the headline suggests (ai2027-tracker.com). The hardest tasks — novel exploit chains, multi-step vuln research — remain largely unsolved.

All four saturate on the same timeline for the same reason: the underlying challenges are public CTF writeups from 2022–2024 competitions, near-certain to be in every frontier model's pretraining corpus, and none of the four attempt decontamination. A benchmark built on public writeups has a shelf life of one to two model generations — the load-bearing fact for anyone thinking about what to build next.

#Real-vulnerability benchmarks: why they keep discriminating

Four benchmarks moved past CTF writeups into live or historical open-source vulnerabilities, and all four still separate model generations cleanly, because their task pools are real, mostly non-public bugs rather than a fixed public set.

CyberGym (UC Berkeley RDI) is the clearest case. At publication in mid-2025, top model/scaffold combinations reached only ~20% success reproducing a real vulnerability from its description. By November 2025, Claude Opus 4.5 scored 50.63% pass@1 across 1,505 tasks — more than 2.5x in under half a year (arXiv:2506.02548). It draws on 1,507 real vulnerabilities across 188 projects, and the team reports finding 34 genuine zero-days and 18 historically-incomplete patches as a side effect of building it. It is also the clearest case of an academic benchmark a frontier lab cites as gating evidence — Anthropic runs it as a named RSP evaluation.

CVE-Bench is two unrelated papers sharing a name. The offense version (UIUC, arXiv:2503.17332) asks an agent to exploit 40 critical CVEs in sandboxed web apps — best agent resolved ~13% at launch, and OpenAI now runs it internally as a recurring regression check on GPT-5.5. The defense version (NAACL 2025, aclanthology.org/2025.naacl-long.212) asks an agent to patch 509 vulnerabilities across 120 repos and four languages — best system repaired only 21%. A third, unrelated "LiveCVEBench" adds to the confusion. None of the three is close to saturated.

BountyBench (Stanford) is the most policy-legible: it scores the full detect-exploit-patch lifecycle in dollars, using 25 real systems and 40 real bug-bounty-listed bounties worth $10–$30,485 each (arXiv:2505.15216). Its best finding isn't a leaderboard topper — it's a clean cross-lab split, OpenAI's Codex CLI stack dominating patching (90%, $14,422 realized) against Anthropic's Claude 3.7 Sonnet Thinking dominating exploitation (67.5%). Dollar-denominated scoring is the single most legible metric format in this landscape.

AIRTBench (Dreadnode) is a different niche — AI attacking AI, not traditional infrastructure. Its 70 challenges target prompt injection, model extraction, and inversion; Claude 3.7 Sonnet leads at 43/70, and frontier models are strong on prompt injection (~49% average) but weak on system exploitation and inversion (under 26%) (arXiv:2506.14682). It isn't cited in any system card yet, but is structurally sound for the same reason CyberGym is: novel, non-public material that can't be memorized from a writeup.

#Defensive benchmarks: the honest state

Offense has six benchmarks and five internal frontier-lab suites. Defense has one credible open effort: CyberSOCEval, published by Meta and CrowdStrike in September 2025, revised that November (arXiv:2509.20166). It covers two SOC-relevant domains — malware analysis and cyber threat intelligence reasoning — built from real telemetry-style tasks, extending CyberSecEval's methodology into the blue-team side for the first time at any real scale.

Its most important finding is not a leaderboard number: reasoning models with test-time scaling do not get the same performance boost on SOC tasks that they get on math and coding. That's a specific, falsifiable result — the single biggest driver of 2025–2026 offensive-eval gains doesn't transfer to defense. It's too early (as of August 2026) to see CyberSOCEval cited in a system card, but it's the closest thing the field has to "CyberSecEval for blue team," and it's the bar the rest of the defensive-eval landscape has to clear.

Below CyberSOCEval, the layer thins fast: CTIBench and CTIArena/CTIConnect for threat-intel reasoning, AttackSeqBench for ATT&CK-sequence comprehension, a 2026 wave of reverse-engineering and malware-attribution papers, and vendor blog-post "benchmarks" (Simbian, Elastic) that aren't independently reproducible or peer-reviewed. No dedicated, widely-cited academic incident-response benchmark exists at all. See AI SOC and detection companies for what vendors selling into this gap actually claim.

The defensive RL gyms — CyberBattleSim (Microsoft, 2021), CybORG (with a 2024 "CybORG++" refresh), and CAGE Challenge 1–4 (TTCP, up through CAGE 4's full enterprise-scale multi-agent environment) — are a genuinely mature, decades-adjacent tradition, but measure something categorically different: narrow RL policies in abstracted network-graph environments, not general-purpose LLM reasoning. The two communities have only started to converge in 2025–2026 papers that plug an LLM in as the policy inside a CybORG or CAGE environment. Treating CAGE scores as comparable to Cybench or CyberSOCEval scores is a category error.

#Frontier labs' internal suites, and the government co-validation bar

The benchmarks that actually gate deployment decisions aren't public. Anthropic's Claude Opus 5 card names five internal evals — ExploitBench, OSS-Fuzz (ARVO-lineage, 830 entry points across 228 projects), Firefox 147 (joint with Mozilla, full-exploit rate jumped from 8.8% on Opus 4.8 to 52.4% on Opus 5), CyScenarioBench, and ExploitGym — none independently runnable by outsiders. OpenAI's GPT-5.5 hub names an internal Cyber Range (93.33% combined pass rate) and VulnLMP (multi-day vulnerability-research campaigns). DeepMind's Gemini 3 Pro card names only its own "Key Skills Benchmark" v1/v2 — it solved 11/12 hard v1 challenges but 0/13 on the harder end-to-end v2 set, tripping an alert threshold without reaching the full Critical Capability Level — and doesn't name Cybench, CyberGym, or CVE-Bench at all, the most internally-opaque of the three.

What all three labs share is external co-validation: UK AISI and US CAISI results, cited by name inside the labs' own system cards, now carry more evidentiary weight than the labs' self-run numbers. UK AISI's cyber-range testing found Opus 5 "capable of attacking small enterprise networks with weak security," and separately found that open-weight models trail the frontier by four to seven months on cyber capability, naming GLM-5.2 and DeepSeek V4-Pro — worth cross-referencing against Open-weight security models. OpenAI's GPT-5.5 disclosure cites UK AISI (90.5% pass@5 on expert-level tasks, including a solved 32-step corporate-network simulation) and third-party evaluator Irregular. This is the real adoption bar, and it's a government-relations problem as much as a research problem: a benchmark or evaluator picked up by a national AI safety institute becomes load-bearing infrastructure for every subsequent frontier release; nothing academic-only reaches that status. See The US picture for the institutional picture US CAISI sits inside.

#The systemic problems

Five problems recur across nearly every benchmark here, and they compound each other:

• Contamination. Every CTF-derived benchmark draws on public, writeup-rich competition problems, and none attempt decontamination. The field's tacit admission: a 2026 wave of papers explicitly brands itself "contamination-free" (LiveCVEBench, arXiv:2608.11469).
• Self-reported, non-reproducible scores. The NYU-CTF leaderboard flags its own numbers as self-reported. Most system-card cyber figures run entirely inside labs on undisclosed prompts, scaffolds, and subsets, making cross-lab comparison unreliable even when the benchmark name is identical.
• Scaffold variance dominates the score. BountyBench shows identical models, wrapped in different agent scaffolds, producing wildly different dollar outcomes. AutoPenBench shows a 21% autonomous vs. 64% semi-autonomous split on the same model. A benchmark result measures the harness as much as the model — see How these systems are actually built.
• Cost per run. Long-horizon agentic benchmarks — CyberGym's 1,507 tasks times multiple rollouts, GPT-5.5's multi-day VulnLMP campaigns — are expensive enough that only frontier labs and well-funded academic groups can run full sweeps, pushing the field toward trusting self-reported numbers by default.
• Environment drift. Live-target and real-CVE benchmarks depend on Docker images and real web-app state that drift over time. CVE-Bench's own changelog shipped a "breaking feature change" in January 2026 — even a well-maintained harness needs regular breaking fixes to stay valid, a maintenance cost nobody prices into "we built a benchmark."

#What it takes to get a benchmark adopted

Cross-referencing what CyberGym, CVE-Bench, and Cybench got right against what CyberSecEval, InterCode-CTF, and dozens of one-paper benchmarks got wrong, five factors recur — and map onto what made SWE-bench the reference point every frontier coding model now reports against [unsourced]:

1. A runnable, Dockerized, reproducible harness shipped with the paper, not just a dataset. CVE-Bench's Apache-2.0 repo and Cybench's harness are the two most-reused pieces of infrastructure in this survey.
2. A continuously extensible, real (not writeup-contaminated) task source. CyberGym's jump from ~20% to 50%+ solved is specifically because its task pool is live, mostly-unpublished vulnerabilities — why Anthropic keeps citing it while CTF sets saturate in a year.
3. A single, policy-legible headline number. "50.63% pass@1 on 1,505 real vulnerabilities" is parseable by a safety team or a journalist; vague multi-axis rubrics — early CyberSecEval's split risk axes — correlate with the ones that went dormant.
4. Third-party or government co-signature within months, not years. The step nearly every benchmark here skips, and the one that actually determines whether a system card cites you.
5. Frontier-lab buy-in at launch, not just an academic audience. Cybench and CyberGym both had near-immediate pickup; benchmarks without it — 3CB, most 2026 malware/RE papers — stay academically cited but invisible in system cards.

Stated plainly: to appear in a frontier system card, a new benchmark needs a Docker-reproducible harness on day one, tasks from real and continuously-refreshable material, one clean top-line metric, and a government AI safety institute or a lab's own safety team rerunning it independently within months. Miss any one and it stays a paper, not infrastructure — relevant to anyone weighing a benchmark against a product as the Go to market motion, since it's a multi-quarter relationship-building exercise with UK AISI or US CAISI, not a ship-and-forget artifact.

#The gap

The specific, citable hole: no CyberGym-scale, continuously-refreshed benchmark scores the full defensive loop — triage, root-cause attribution, scoping, remediation a human IR lead would accept — the way BountyBench scores the full offensive loop. CyberSOCEval splits malware analysis from CTI reasoning; nothing chains them into one incident. Separately: no widely-adopted "review this diff for vulnerabilities" leaderboard exists. The closest candidates — SWE-PRBench (general code-review quality, not security-specific), a March 2026 paper on contextual bias in LLM security code review, and a 2025 repo-level vulnerability-detection benchmark — are each partial, and none has a maintained leaderboard.

Both gaps share CyberGym's insight: build from real, continuously-sourced material instead of a fixed public set, score with one legible number instead of a rubric. Given CyberSOCEval only shipped in September 2025 and no August 2026 system card reports a defensive score with anything like ExploitBench's specificity, defense sits roughly where offense sat in mid-2024 — one serious benchmark, no continuously-refreshed real-incident pool, no dollar-denominated headline metric, not yet cited as gating evidence anywhere. A twelve-to-eighteen-month gap, not a permanent one.

#What this means for us

• Do not plan the company around releasing a benchmark. None here generate revenue for their makers; they generate citations and, occasionally, RSP-eval status — a relationship with a lab's safety team, not a customer.
• The one real gap — full-loop, cost-denominated defensive benchmarking — needs a UK AISI or US CAISI relationship to matter, and that relationship-building is better spent on product credibility than a standalone eval. See Where the gaps actually are for how this compares to other unclaimed positions.
• If we publish an eval at all, keep it narrow and attached to a real product claim ("our agent scores X on a diff-review benchmark we built and open-sourced"), not a standalone research contribution — the CyberGym/BountyBench playbook, not CyberSecEval's.
• CyberSOCEval's test-time-compute finding is a genuine warning: don't assume a bigger reasoning budget on SOC-style tasks buys the same gains it buys on coding. Validate this ourselves before betting the roadmap on it.
• Contamination and scaffold variance mean any benchmark score we quote — including our own — should be treated skeptically until we know the harness, subset, and whether it's self-reported. See Verification ledger for the discipline this applies to exactly this kind of number.
• Watch AIxCC's descendants closely (AIxCC: the closest thing to a proof) — the closest thing to a full-loop, cost-denominated, third-party-witnessed defensive benchmark that exists, even though it's framed as find-and-fix rather than SOC triage.