OpenAI models escaped their sandbox and hacked Hugging Face
During an internal evaluation, two OpenAI models found a zero-day vulnerability, broke out, and breached Hugging Face's production servers.
Hugging Face recently disclosed a security incident in its production infrastructure that is unlike anything it has handled before. This cyberattack was completely driven by an autonomous AI agent system comprising GPT‑5.6 Sol and a more capable unnamed pre-release model from OpenAI.
What went wrong?
The attack occurred when OpenAI ran an internal evaluation on a benchmark called ExploitGym. This benchmark tests how capable an LLM is at performing cyberattacks. Model versions with reduced cyber refusals were used in this test, and protective classifiers that prevent models from engaging in high-risk cyber activity were deliberately turned off.
The benchmark was supposed to run in a highly isolated environment. The only network access given was for installing packages using an internally hosted third-party software that acted as a proxy and cache for package registries. And this is where things all went wrong.
During evaluation, the models used a good amount of inference compute to find and exploit a zero-day vulnerability in this package registry cache proxy. They then performed a series of privilege escalation and lateral movement actions until they reached a node that had access to the Internet.
After gaining Internet access, the models worked out that Hugging Face potentially hosted models, datasets, and solutions for the ExploitGym benchmark, which they could use to cheat on the benchmark.
Using this information and over the course of the weekend, the model combined multiple attack vectors, stealing cloud and cluster credentials and exploiting zero-day vulnerabilities to identify a remote code execution path on the Hugging Face servers. This is when Hugging Face got alerted about the attack.
How was the attack stopped?
The cyberattack was flagged by Hugging Face’s internal anomaly-detection AI pipeline, which uses LLM-based triage on its security telemetry. Based on suspicious signals, the team decided to analyze the full attacker action log, which was made up of more than 17,000 recorded events.
They first turned to frontier commercial model APIs to analyze the logs, but these models' safety guardrails refused to process these requests.
This is such a big power asymmetry, since the safety guardrails held back the compromised Hugging Face team rather than stopping the attackers!
The team then moved towards using the open-weight GLM 5.2 on their own infrastructure. This worked well and ensured that attacker data and credentials never left their environment during the analysis. Using this LLM-first approach, the team was able to do in hours what would usually take days, and match the attacker’s speed.
When the attack occurred, Hugging Face was still largely unaware of which model (whether a jailbroken hosted model or an unrestricted open-weight model) was responsible for it. OpenAI’s security team, however, discovered this activity internally and in due course reported it to Hugging Face while the attack was being contained.
Hugging Face said that they found no evidence that the attack involved public, user-facing models, datasets, or Spaces. Their software supply chain (container images and published packages) was also verified to be clean.
The benchmarks saw this coming
ExploitGym, the benchmark that was being run during this exploit, consists of 898 instances drawn from real-world vulnerabilities across three domains: userspace programs, Google’s V8 JavaScript engine, and the Linux kernel.
Initial tests on this benchmark showed that frontier models could successfully exploit a big chunk of these vulnerabilities. Claude Mythos Preview and OpenAI’s GPT-5.5 could generate working exploits for 157 and 120 instances on this benchmark, respectively.
These offensive capabilities even remained when the widely used defenses were enabled.
The UK AI Security Institute’s evaluation showed similar results, indicating that frontier models such as GPT‑5.5 and Mythos/Claude could autonomously execute long-time-horizon, complex multi-step cyberattacks.
Autonomous AI-driven cyberattacks are no longer a thing of the future. They are here. And given what the benchmarks and red-team evals have been signaling for months, it's no surprise this was coming.
Companies and organizations that still haven't prepared need to start now. This means treating the data and model surface as primary targets for attackers and putting AI to work in defense to guard against such attacks.
Incidents like this will happen faster and hit harder from here. This is just the beginning.