← AI glossary

AI glossary · Basic · AI security · Agents · Data & evaluation

ExploitGym

ExploitGym is a benchmark that measures whether AI agents can turn already-known security vulnerabilities into actual attack effects. It was released as an arXiv paper in May 2026 and consists of 898 tasks drawn from real vulnerabilities.

About this term
Level
Basic · Technical terms you often meet in AI news
Added
2026-11-14
In plain words It is similar to a 'mock lock-picking test ground' that checks whether you can actually open a door when all you know is that the lock has a flaw. However, the real test ground sits inside isolated containers, and because the results measure attack capability, they also serve as data for gauging risk on the defense side.

What it is

In security, a vulnerability is a weak spot in a program. On its own, it is not yet an attack. Real damage happens only when the weakness is used to go as far as reading files or executing code without permission, and this process is called exploitation. The paper's authors considered this capability very important but not properly evaluated so far, so they built ExploitGym. They aimed for an evaluation tool that is large-scale, diverse and close to real-world conditions.

How it works

In each task, the agent receives one program input that triggers a vulnerability. It must then expand this input step by step into a working exploit. The tasks come from real vulnerabilities in three areas: ordinary user programs, Google's V8 JavaScript engine and the Linux kernel. By changing the configuration of the same vulnerability, such as turning security protections on or off, you can also separately see how much defenses affect the success rate. Every configuration is packaged in a container (an isolated execution environment), so anyone can rerun it under the same conditions.

Flow of a single ExploitGym taskReal vulnerability → Vulnerability-triggering input; Vulnerability-triggering input → AI agent; AI agent → Working exploit?(Success judgment); Container environment → AI agent(Execution environment); Security protections → Container environment(Configuration change)Real vulnerabilityUser programs · V8 ·Linux kernelVulnerability-triggering inputAI agentExpand input step bystepWorking exploit?File access · codeexecutionContainerenvironmentReproducible isolatedexecutionSecurityprotectionsIsolate impact byturning on/offSuccess judgmentExecution environmentConfiguration change
Flow of a single ExploitGym task Read from left to right. An agent that receives an input triggering a vulnerability expands it into an actual attack effect inside an isolated container, and success or failure is judged.

Why it matters

According to the paper, exploitation is still hard, but top models actually exploited a substantial number of vulnerabilities. The strongest configurations were Anthropic's Claude Mythos Preview and OpenAI's GPT-5.5, which produced working exploits on 157 and 120 tasks, respectively. Even with widely used defenses turned on, success rates were not negligible. Since then, as AI companies have released models specialized for cybersecurity, they have increasingly cited ExploitGym scores as evidence of performance.

Things to know

Exploitation capability is a dual-use technology. Defenders can use it to prioritize patches or verify risks, but it also lowers the barrier to entry for attackers. So the score is read not only as an indicator of 'how useful a model is' but also of 'how dangerous it could become.' Scores can also vary greatly depending on protection settings and task scope, so when comparing numbers from different sources, first check whether the conditions are the same. Some figures circulating in the community are unverified.

Example

On this site, ExploitGym was mentioned in news about OpenAI's cybersecurity model 'GPT-6 Cyber.' According to reports, OpenAI said that in an ExploitGym comparison GPT-6 Cyber scored 48.5% and the previous model, GPT-5.6 Cyber, scored 32.1%. The model was described as finding vulnerabilities, producing fixes, and being trained to operate only within authorized scope. Around the same time, a claim circulated in the community that an internal OpenAI model scored 77.0%, but this is unverified hearsay and the measurement conditions are unknown. When security teams look at such scores, it is best to first check which area and protection settings the numbers were measured under.

Where it came up on this site

Related terms

References