Post-training playbook

Agentic coding RL is a solved-enough engineering problem: five-plus teams have reproduced "container + tests + GRPO-family algorithm → 40-60% SWE-bench Verified" within a year. Security-specific RL is not — the literature is mostly pre-RLVR steering methods or benchmarks establishing headroom, not training recipes. The crux isn't compute or algorithm choice; it's verifier design, and most obvious reward signals for security tasks are gameable in predictable ways. This page covers prior art, the verifier table, environment-build costs, stack choice, real 2026 pricing, the legal wall around the cheap shortcut, and a team/timeline plan.

#Prior art: the agentic coding RL lineage

The direct methodological ancestor for defensive security — "patch a codebase, run tests, pass/fail" maps almost exactly onto "patch a vulnerability, run the regression suite plus PoC, pass/fail."

System	Base / scale	Reward	Compute	Result	Note
SWE-RL (Meta)	Llama3-SWE-RL-70B	diff-similarity, not execution	not disclosed	41.0% SWE-bench Verified	cheap at scale, but a proxy reward
SWE-Gym	2,438 executable Python tasks	unit tests + reranker model	—	32.0%, +19pt over baseline	CC-BY-4.0
SWE-smith	50,000 tasks, 128 repos, procedural test-breaking	held-out tests	—	40.2% pass@1 (32B)	template for generating tasks without hand curation
R2E-Gym	8.7K+ procedural environments	hybrid: execution + LLM-judge	—	51%	pure execution "saturates," pure judges are style-biased — combine both
DeepSWE	Qwen3-32B, GRPO++	pass held-out tests in 5-min budget or zero	64 H100 × 6 days = 9,216 H100-hrs	59% (k=16)	"compact filtering" masks trajectories hitting the step limit, or the model games episode length
Kimi-Dev-72B	72B, Docker-based	all-or-nothing: full suite passes	not disclosed	60.4%	stricter, harder to partial-credit-game

What the field converged on: procedural task generation over hand curation, a hybrid execution+judge verifier over either alone, explicit handling of episode-length reward hacking. DeepSWE's 9,216 H100-hour, ~$35K RL-stage figure is the single most useful compute anchor in this document.

#Security-specific prior art, and how thin it is

Pre-RLVR work exists but doesn't touch agentic vuln-hunting: SVEN (2302.05319, 2023) steers a frozen CodeGen-2.7B toward secure completions via prefix vectors, no weight updates, secure-code rate 59.1%→92.3% — controllable decoding, not RL. SafeCoder (2402.09497, 2024) is security-centric SFT, ~30% security improvement while preserving utility. HexaCoder (2409.06446, 2024) uses an oracle to synthesize secure/insecure pairs, up to 85% reduction in vulnerable generations — the closest analog to SWE-smith's procedural pattern, but it targets code-generation-time security (don't write the bug), not agentic remediation (fix an existing bug), a materially easier problem.

The one clean RLVR precedent is Minerva (2602.00513) — CTI mapping (CVE→CWE, ATT&CK technique, CVSS-vector, threat-actor alias resolution) across 16 tasks / 32,000 instances with fully deterministic, programmatic verifiers. MinervaRL augments GRPO with a hardness-gating trick — when every rollout fails on a hard example, it briefly reveals the gold label to elicit a rationale, verifier-filters it, distills it back via an EMA teacher, fixing RLVR's classic zero-gradient-on-hard-examples problem. Result: +15.8 points over base, +4.3 over vanilla GRPO across four backbones, trained in only 500 steps. It works because CTI-mapping has real deterministic verifiers — that's the pattern to copy, not the task.

Everything else is a benchmark, not a recipe: CyberGym (1,507 real vulns, 188 projects, best success 20%; building it surfaced 34 real zero-days and 18 historically-incomplete patches as a side effect); CAIBench (10,000+ instances; knowledge questions saturate ~70%, multi-step Attack-and-Defense drops to 20-40%, scaffold choice alone producing up to 2.6x variance on the same model); Exploit-CVE-Bench (13% best, 2503.17332) and Repair-CVE-Bench (~21% best, NAACL 2025); CyberChainBench (37.5% detection, 43.7% exploitation, only 23.4% patching, 2606.26216). Patching consistently lags detection and exploitation — plan for remediation as the hardest sub-skill. The most relevant public precedent for automated patch generation at scale is AIxCC: the closest thing to a proof: 63 synthetic vulns, 86% found, 68% patched, all seven finalist systems open-sourced (Team Atlanta, Trail of Bits, Theori placed) — worth studying as an engineering reference.

#The crux: verifier design

For each candidate reward signal — what it is, how it gets gamed, and whether it belongs as a primary signal.

Verifier	Signal quality	How a policy exploits it	Verdict
PoC exploit executes / crashes target	High, binary	Triggers any crash, not the specific vuln class	Clean only paired with crash-signature matching against the CVE's known crash site
Sanitizer fires (ASan/MSan/UBSan)	High	False-positives on benign UB; model farms harmless UB the sanitizer flags	Clean-ish paired with a differential check: fires pre-patch, not post-patch
Test suite passes (post-patch)	Medium	Delete/weaken the failing test, special-case the exact input, reintroduce the vuln via another path — R2E-Gym: execution-only verifiers "saturate"	Needs held-out tests invisible to the policy, plus a diff check test files weren't touched
Fuzzer crash reproduces	High if corpus/harness frozen	Overfits the seed corpus; suppresses instrumentation if it controls build flags	Clean when corpus/harness sit outside the policy's action space
Patch applies + PoC no longer reproduces + regression suite green	High — gold standard	Dodges the specific PoC precondition without fixing the vuln class; or "fixes" by deleting the feature	Pair with a green regression suite and, ideally, a class of PoC variants, not one input
SARIF / static-analyzer match	Low	Satisfies the pattern-matcher without changing runtime behavior — "teaching to the linter"	Usable only as a pre-filter, never primary
LLM judge	Low	Sycophancy/style hacking — verbose, confident patches score higher regardless of correctness	Only usable combined with an execution-based signal

The general principle: reward signals in security are clean in proportion to how close they are to an independent, frozen, policy-invisible oracle that actually re-executed the exploit or regression suite in an isolated environment the policy cannot edit. Every step away from that reopens a specific, predictable exploit. A dedicated verifier-fuzzing paper, "Before the Model Learns the Bug", fuzzes RLVR verifiers across math/JSON/code domains and concludes bluntly: if the verifier is wrong, optimization will learn the bug. No equivalent study exists for security-RLVR verifiers specifically — assume your own has undiscovered holes and budget red-teaming time against the reward function, not just the trained policy.

The clean signal, stated plainly: "patch applies + PoC no longer reproduces + full regression suite green" as primary reward — binary, sparse, expensive, but clean — shaped with a smaller-weight sanitizer/test-pass auxiliary term for partial credit (Minerva's hardness-gating is the template for the resulting sparsity). Never let SARIF or LLM-judge signals be more than auxiliary.

#Environments: building 10k+ verifiable security tasks

Concrete precedents to build on rather than reinvent. ARVO (GitHub, v3.0.0, Oct 2025) reproduces OSS-Fuzz vulnerabilities as Docker images — a vulnerable-commit image the recorded PoC triggers, a patched-commit image it doesn't. Current release: ~6,138 reproducible security issues, patch-commit identification >80% correct — ready-built infrastructure for "PoC no longer reproduces" verifiers at scale. CyberGym-E2E (2606.04460) extends CyberGym to the full lifecycle — discovery → PoC → patch — across 920 vulnerabilities, 139 projects, built with an "automated, agent-enhanced pipeline" that itself uses agents to convert raw vuln records into runnable environments: the most direct precedent for building verifiable tasks without an army of human curators. It's smaller by task count than SWE-smith's 50K for a comparable project count, consistent with real CVEs with real PoCs being harder to automate than synthetically breaking an arbitrary test.

The harness needs: a containerized immutable base image per project/commit; a frozen PoC/test corpus outside the policy's action space; snapshot/restore between rollout steps (rebuilding a C/C++ project from scratch each step is prohibitive — expect 10-60 seconds per build/test cycle, the dominant cost driver of rollout wall-clock, not the LLM forward pass); heavy environment parallelism relative to GPU count, since a 100-step rollout can take 15-100 minutes wall-clock with the policy idle while the environment executes (why verl and SkyRL foreground async rollout-vs-training architectures); and budget for flake rate as unsolved, since no paper surveyed here published a number for it.

Cost estimate: budget 2-4 engineer-months to stand up the container/harness/snapshot infrastructure once, then roughly 0.5-2 engineer-days per project (not per task instance) to onboard a new codebase, after which procedural generation within it is close to free. Cost scales with codebase count, not task count. A 10k-task, ~150-codebase environment at CyberGym-E2E's scale is a 1-2 engineer-quarter effort, built on ARVO and OSS-Fuzz's own harnesses rather than from scratch.

#Stack recommendation for 2026

Framework	Agentic/long-rollout fit	Status
verl (ByteDance)	multi-turn + tool-calling native	Most proven: scales to 671B models, used by ByteDance/Qwen/Tsinghua/Berkeley. Supports PPO, GRPO, GSPO, DAPO, RLOO. Default for a 2026 run.
SkyRL	purpose-built for SWE-bench-style tasks	v0.2.0, Apr 2026; smaller community, exactly fit-for-purpose; Tinker-compatible
OpenRLHF	explicit multi-turn reset/step mode	mature (9.6k stars), scales to 70B+
TRL (Hugging Face)	agentic/tool-use RL not first-class yet	v1 just shipped — SFT/warm-start tool, not the RL engine
Tinker	managed API, you own the loop	usage-based ($/M tokens); no cluster to run
Unsloth / Axolotl / LlamaFactory	single-turn SFT/LoRA	SFT/warm-start stage only

Bottom line: verl for your own cluster and broadest algorithm support; SkyRL if long-horizon tool-use is your primary use case; Tinker if you'd rather pay per-token than run infra; Unsloth/TRL for SFT warm-start regardless. All of this stack is The open-source stack, which matters for auditability of a security-critical training pipeline as much as for cost.

#Cost tables: 2026 GPU pricing and real budgets

#GPU hourly rates (fetched live, Aug 2026)

Provider	H100 on-demand	B200
Nebius	$3.85 ($2.15 preemptible)	$7.15 ($3.95 preemptible)
Together AI	$3.99	$8.19
Modal	$3.95	$6.25
RunPod	$3.29 (SXM) / $2.89 (PCIe)	$6.79
Lambda	$3.99	$6.69 on-demand / $8.87-9.86 reserved
CoreWeave	~$6.16 (8-GPU node)	~$8.60; spot ~50-60% of on-demand

Sources: Nebius, Together, Modal, RunPod, Lambda, CoreWeave. H100 on-demand clusters around $3-4/GPU-hr across four of six providers; CoreWeave prices higher, reflecting an enterprise/reserved-first model. B200 runs 1.6-2x the H100 rate for meaningfully more throughput. Nebius is cheapest at every tier here.

(a) SFT of a 7B-32B model: no provider publishes a turnkey number; built from GPU-hours. A LoRA SFT run on a 32B model over a few-hundred-thousand-example set needs 8-16 H100-class GPUs for 1-3 days: at Nebius, 8 GPUs × 48 hrs ≈ $1,480; at the high end, 16 × 72 hrs ≈ $4,435. Realistic budget: $1,500-$6,000, before the (often larger) cost of the instruction dataset itself.

(b) An agentic RL run comparable to DeepSWE: 9,216 H100-hours is the anchor — Nebius ≈ $35,500, Together ≈ $36,772, CoreWeave on-demand ≈ $56,770. Realistic budget for one 32B-class run: $30,000-$60,000, not counting SFT warm-start, environment engineering, or the 2-4x compute typically burned on failed runs before the keeper — budget $100K-$200K total for one released checkpoint.

(c) A realistic 12-month budget for a 5-person lab

Line item	Low	High
Salaries (2 RL/infra, 1 security-domain, 1 data eng, 1 lead/PM, fully loaded)	$900K	$1.6M
Environment/harness engineering	$60K	$150K
SFT warm-start runs	$15K	$50K
Agentic RL runs (3-6 attempts)	$150K	$400K
Eval infra + repeated benchmark runs	$20K	$60K
Serving for dogfooding/pilot	$10K	$80K
Frontier-API costs for data gen/judge (subject to legal caveats below)	$20K	$100K
Storage, tooling, misc	$15K	$40K
Total (excl. salaries)	~$290K	~$880K
Total (incl. salaries)	~$1.2M	~$2.5M

The naive "just the training run" number quoted in blog posts ($30-60K) understates true program cost by roughly an order of magnitude once iteration, environment engineering, and evaluation are counted. See Unit economics and the compute bill for how this sits against the rest of the cost stack and Who funds this and at what price for round-size implications.

Serving cost — the number most cost models miss: a 200-turn agentic rollout at 2-8K tokens/turn is 400K-1.6M tokens per episode — one long investigation can burn as much inference as a book. Serverless open-model pricing: gpt-oss-20B runs $0.05/$0.20 per 1M input/output tokens; Llama 3.3 70B runs $1.04/$1.04 per 1M (Together). A 1M-token episode costs roughly $1-2 on a 70B model or $0.10-0.25 at gpt-oss-20B scale — plausibly 5-20x cheaper per episode than a frontier closed-model API at the same volume, but the crossover only pays off once the $100K-$200K training fixed cost is amortized across real production volume.

#Legal: what the ToS actually rule out

All three frontier labs (see What the frontier labs do themselves) use near-identical language, worth quoting precisely. Anthropic: "Customer may not and must not attempt to (a) access the Services to build a competing product or service, including to train competing AI models... except as expressly approved by Anthropic" (commercial terms). OpenAI prohibits "Use Output to develop models that compete with OpenAI" alongside a bar on reverse-engineering the Services. Google Gemini API: "You may not use the Services to develop models that compete with the Services... You also may not attempt to reverse engineer, extract or replicate any component of the Services, including the underlying data or models" (terms).

This is a contractual restriction, not a copyright claim — enforceable by account termination and breach-of-contract action, not (under current US law) copyright. It is not theoretical: in February 2026 Anthropic publicly accused Chinese AI labs, DeepSeek among them, of using fraudulent accounts to generate millions of Claude conversations to train their own models, reported by TechCrunch — a live, public enforcement precedent, not a hypothetical.

The safer path is what SWE-smith, R2E-Gym, DeepSWE and Minerva all did: train on real, historical, publicly-licensed data — GitHub PR/issue history, OSS-Fuzz/ARVO's actual CVEs and patches — with execution-based ground truth rather than distilled frontier-model outputs. That's also the technically cleaner reward per the verifier table above — legally safer and more rigorous point the same direction.

#The honest question: does post-training beat prompting + scaffolding?

It does not, on raw capability, for most teams, from scratch. CAIBench found that for the same model, scaffold choice alone produces up to 2.6x variance on Attack-and-Defense scores — if scaffolding dominates outcome variance that strongly, first-dollar investment belongs in How these systems are actually built, not weight updates. Every agentic security benchmark surveyed here is nowhere near saturated even with frontier models plus good scaffolding (CyberGym ~20%, Exploit-CVE-Bench ~13%, Repair-CVE-Bench ~21%, CyberChainBench 23.4% patching, see Cyber benchmarks and evals) — a small post-trained model starting from a weaker base starts from a worse position on a hard task, not an easy one it's catching up on. DeepSWE, a heavily RL-trained 32B model with $30-60K of compute, reaches 59% SWE-bench Verified on coding — a domain with a far more mature verifier ecosystem than security has — and frontier models with good scaffolding are reported competitive with or ahead of that, no custom training required. And the cheapest way to close a capability gap — distillation from frontier-model outputs — is exactly what the ToS language above rules out.

Where a small post-trained model genuinely wins, quantified:

• Cost at volume: roughly $0.10-2 per 200-turn episode on cheap serving infra vs. a materially higher frontier-API rate at the same token volume — above some real production threshold (hundreds-to-thousands of episodes/day), a post-trained small model is strictly cheaper per investigation.
• Latency: structurally, dedicated inference has no shared-capacity queueing under load, and per-turn latency compounds linearly across a 200-turn loop — a 200ms-per-turn reduction is 40 seconds saved per episode, non-trivial for interactive incident response.
• Air-gapped/on-prem deployment: close to a hard requirement for classified environments, regulated critical infrastructure, and financial-sector SOCs with data-residency constraints — no frontier API can serve these at all, independent of capability. This is the single strongest, least-arguable case for post-training: it's a deployment gate closed frontier APIs cannot pass regardless of how good they are. See Is frontier-lab gating a real wedge?.
• No-refusal / security-specific calibration: frontier safety training is tuned against a broad misuse population overlapping heavily with legitimate defensive workflows (a PoC to verify a patch, malware analysis, explaining an exploit chain to a SOC analyst). A domain-specific model can be calibrated more precisely — real, but partly fixable via scaffolding too, so weight it below the air-gap argument.
• Structured sub-tasks: Minerva's CTI-mapping precedent hit +15.8 points over base on a tiny compute budget. For narrow, well-verified, high-volume sub-tasks, small RL-tuned models are a clearly good, cheap, low-risk investment, distinct from the harder full-agent case.

#Minimum viable team and timeline

Treating "respectably on a public benchmark" as competitive with current open-model SOTA (20-40% success on CyberGym/Repair-CVE-Bench/CAIBench Attack-and-Defense) rather than beating frontier-plus-scaffolding:

Team (5-6 minimum): one RL/training-infra lead owning the verl/SkyRL stack and GRPO-family tuning — the hardest role to hire and the single biggest determinant of success, per the DeepSWE/Kimi-Dev/R2E-Gym precedent of teams with deep in-house RLVR experience; one to two environment/harness engineers owning the container/snapshot/verifier pipeline; one security-domain expert who owns verifier-correctness judgment — without this role you ship a reward-hacked model and don't notice for months; one data/eval engineer; one lead/PM who can also do general ML engineering. See Talent: the actual constraint for how scarce the RL-infra-lead profile is right now.

Timeline: Months 1-3, stand up harness infra on ARVO/OSS-Fuzz, get 500-1,000 verified tasks running on the clean verifier, stand up verl or SkyRL, run a small pilot on a Minerva-style narrow sub-task to validate the pipeline cheaply. Months 4-7, scale toward 5,000-10,000+ tasks, run the first full agentic-RL pass on a 7B-14B model, expect 3-5 failed runs before a usable checkpoint. Months 8-10, scale to 32B, run the keeper RL pass (budget $30-60K, expect 2-3 attempts), run full eval against CyberGym/CVE-Bench/CAIBench. Months 11-12, harden against reward hacking found during eval, pilot the air-gapped/on-prem deployment that's the strongest economic case for the program, publish results.

Realistic 12-month outcome for this team and $1.2M-$2.5M all-in: a 7B-32B model competitive with open SOTA on narrow verified sub-tasks, and in the 10-40% band on full agentic patching — not one that beats a frontier model with good scaffolding on raw capability, but one deployable air-gapped, cheaper per-episode at volume, and calibrated for defensive workflows. See The first 90 days for how the first quarter nests into the lab-building plan, and The three ideas, judged for how this rolls up against the rest of this research's build-vs-buy calls.

#What this means for us

• Do not attempt full agentic security-agent RL as a first project. Start with a Minerva-style narrow, deterministically-verified sub-task — cheap, fast, and it validates the pipeline before committing real budget.
• The verifier table is the actual product decision: commit early to "patch applies + PoC no longer reproduces + regression suite green" as the primary reward and treat SARIF/LLM-judge signals as auxiliary only, or the program will quietly reward-hack itself.
• Build the environment layer on ARVO (6,138 reproducible OSS-Fuzz vulns) rather than from scratch — this turns a multi-year infrastructure problem into a 1-2 engineer-quarter one.
• Budget $1.2M-$2.5M for a real 12-month program with a 5-6 person team, not the $30-60K "one training run" number that circulates in blog posts — salaries and iteration dominate, not the headline RL compute figure.
• Do not build the training corpus from frontier-API outputs if the resulting model competes with that vendor — the Anthropic/DeepSeek episode is a live enforcement precedent, and it also points toward the technically cleaner ARVO/historical-CVE path anyway.
• The credible pitch to investors and buyers is deployment constraints and unit economics at volume (air-gapped SOCs, cost per episode at scale), not "beats Claude/GPT at security" — no one has shown that on agentic tasks, and claiming it invites an easy benchmark challenge.