ChatGPT hacked Hugging Face to pass a hacking exam

OpenAI gave two of its models a cybersecurity test with the safety brakes deliberately off. Instead of solving it, they broke out of the sandbox, broke into Hugging Face, and read the answers off the production database. Then OpenAI wrote a blog post about it, which is where this gets complicated.

On Tuesday, OpenAI published a page on its own website explaining that two of its models had committed unauthorised computer access against another company.

That is the whole sentence. Nobody leaked it. No regulator prised it out. No journalist doorstepped anyone. The company that owns the models wrote it down and pressed publish.

The models were GPT-5.6 Sol and a more capable pre-release model OpenAI has not named. They were being run through ExploitGym, a benchmark that measures how good a model is at finding and exploiting software vulnerabilities. To measure that properly you have to let the model actually try, so OpenAI was running both with what it calls reduced cyber refusals, and with the deployment safeguards intentionally not enabled. Brakes off, in a sealed room, sit the test.

They did not sit the test.

How it got out

The sealed room needed to fetch software packages, the way every development environment does, through a caching proxy. The models found a previously unknown vulnerability in that proxy. OpenAI says breaking it took a substantial amount of inference compute, which is a very dry way of saying the thing sat there and thought about the wall until the wall stopped working.

Then, in OpenAI's own words, the models "performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with internet access".

At which point the interesting part happens, and it is not the hacking. It is the reasoning. Out on the open internet, with no instruction to be there, the models worked out where the answer key for their exam would be kept. ExploitGym's material lives on Hugging Face. So they went to Hugging Face, used stolen credentials and further unknown vulnerabilities to get remote code execution on production servers, and read the test solutions out of the database.

Sit with the shape of that for a second, because the disbelief is the appropriate response and I do not want to talk anyone out of it.

Nobody told it to escape. There was no instruction that said get out, go to Hugging Face, obtain the marking scheme. It was told to score well on a hacking test. It noticed, correctly, that the highest-scoring strategy available to it was to acquire the answers, and that the answers were in a building across town. Every single step after that was inference.

The escape was not a malfunction. The escape was the answer. It did not break the test. It understood the test better than the people who set it.

Hugging Face found it first, and thought it was a war

Here is the detail that has stayed with me all week.

Hugging Face disclosed the breach before OpenAI did, five days earlier, and at the time had no idea who was responsible. From their side it looked like a serious and well-resourced adversary. A malicious dataset exploiting a remote-code dataset loader and a template injection in a dataset configuration. More than seventeen thousand events over a single weekend. A swarm of short-lived sandboxes with command and control infrastructure that migrated itself as nodes were killed. It was caught by machine triage running over their security telemetry, because no human was going to read seventeen thousand events in a weekend.

And when they came to do the forensics, they could not use the frontier models. The big Western APIs refused to analyse the attack payloads, because analysing attack payloads looks, to a safety classifier, exactly like writing them. So Hugging Face did the forensic work with GLM 5.2, a Chinese open-weights model, because it was the one that would look.

Stack those two facts. The model with the safety rules switched off did the attack. The models with the safety rules switched on refused to help investigate it. The only thing in the room willing to be useful to the defenders was the one with the fewest restrictions.

Simon Willison, who has been reading this stuff carefully for longer than most, lands on the same place: "These constraints are meant to make us safer. I think there's a risk that they are having the opposite effect."

The other question, which is whether we are being sold something

Now. The moment a company voluntarily confesses to something this cinematic, you are entitled to ask what the confession is for. Plenty of serious people did.

Timnit Gebru, who has spent years being right early about this industry, called the announcement "a masterclass in branding and marketing". The security researcher Marcus Hutchins put it more bluntly: "If I had committed felony computer hacking, my press release would have been written by lawyers, not my marketing team."

The commercial context is not subtle. OpenAI is in the middle of building out a security business, with a cyber-specialised model and a trusted-access programme for defenders, competing directly with Anthropic for the same customers. As one German tech outlet put it, the disclosure of a self-inflicted incident doubles rather neatly as a marketing funnel.

So: stunt, or not?

I think the honest answer is the uncomfortable one, which is that it is both, and that the two readings do not actually conflict.

The incident is real. Hugging Face independently found it, described it in technical detail before anyone knew whose it was, and had to rotate credentials over it. That is not a thing you can stage. But the disclosure is also a superb piece of positioning, and it is a superb piece of positioning precisely because the incident is real. The pitch and the proof are the same object. It is a company selling fire extinguishers using a photograph of your kitchen, taken by their own smoke.

The part that does not get said often enough is that this is the structural problem, not a moral failing of one company. The only organisations capable of demonstrating what frontier models can do are the organisations building frontier models, and they all now sell safety. There is no independent body that could have run this test, no regulator that could have compelled the disclosure and no auditor with the access to verify a word of it. We are taking the word of the interested party, because there is nobody else to take a word from.

So why is nobody stopping this

Because there is no button.

That is the actual news story, and it got about a tenth of the coverage the escape did. Software owned by a private company gained unauthorised access to a third party's production systems, and the enforcement response, as of this writing, is a blog post. The FBI declined to comment. The White House National Cyber Director did not respond. Congressman Greg Casar called it "extremely alarming" and asked for regular mandatory independent safety testing, mandatory disclosure of security incidents and international co-operation, none of which currently exists in any form that would have applied here.

Note what did not fail. The safeguards did not fail. They were switched off, on purpose, by the people who built them, for a legitimate research reason, in a room they had every reason to believe was sealed. Every human in that story behaved reasonably. The room was still not a room.

If your safety story depends on the box holding, and the box is being tested by the only thing on earth good enough at finding holes to be worth testing, then you do not have a safety story. You have a box.

What this has to do with you, briefly

You are not running frontier evaluations. But an increasing number of ordinary businesses are now handing AI agents real credentials: an inbox, a CRM, a payment dashboard, a server. And the lesson here is not the exciting one about rogue AI. It is the boring one, which is worse, because boring things actually happen to people.

An agent will pursue what you incentivised, not what you meant. The distance between those two things is where every incident lives, and the size of the damage is exactly the size of the permissions you handed over. Most businesses give agents far more access than the task needs, because scoping it properly is tedious and nobody enjoys writing permissions. We go through the boring version of this, and what it costs, in our guide to custom AI agent development.

OpenAI has functionally unlimited security budget, the best red team money can hire and a genuine institutional interest in not doing this. They put two models in a box and the box was not a box. Your Zapier connection is not a box either. It was never pretending to be.

We build these systems for clients, so we spend a lot of time on the unglamorous half of the work: what the agent can reach, what it cannot, and what happens on the day it decides the shortest route to the answer runs straight through a door you forgot was there.