Over the last several weeks, the world has become aware of several high-profile cybersecurity incidents involving AI. I'll spoil my main conclusions first, then tell you how I got there:
- The novel thing here is that AI models can autonomously chain together several significant tasks to achieve a higher-level goal. Models have been good at finding and exploiting individual bugs in software for a while now, but successfully formulating and executing a plan to conduct several unique attacks - against real systems - to meet a higher objective is new.
- There will be some rough times over the next few years. These recent incidents are a harbinger of things to come. More incidents like these will happen in the near future. We will be forced to figure out how to effectively govern and safeguard AI models that are increasingly capable of both good and destructive things.
- Banning models won't work. In fact, it'll make things worse.
I think these conclusions have already been vindicated. In just the last month we have learned:
- OpenAI models broke out of containment and hacked another AI platform (Hugging Face). The models exploited previously unknown vulnerabilities to do this, and OpenAI was not aware this happened until some time later (we're not sure exactly how long).
- An Australian man asked his AI assistant to see if it could get him into a local gym class that was already fully booked. The AI assistant exploited vulnerabilities in the gym's website to delete another member off the waitlist so he could get in faster.
- While performing their own cybersecurity tests, Anthropic detected several instances of its own models unintentionally operating outside of what was supposed to be a walled-off test environment.
- The AISI (part of the UK government) conducted their own testing and detected AI models trying to perform a supply-chain attack on real open-source software while they were testing the models' ability to solve cybersecurity challenges (though AISI handled this incident MUCH better than OpenAI handled theirs).
In my opinion, the OpenAI / Hugging Face incident from mid-July was the most significant of these incidents and the one we can learn the most from, so this article will focus on that incident in particular.
What happened?
Hugging Face is a web platform that hosts AI models, datasets, and apps. On July 16th, 2026, they publicly disclosed they were hacked and that the attack "was driven, end to end, by an autonomous AI agent system." In other words: Hugging Face was hacked by AI.
Five days later, on July 21st, OpenAI dropped a post addressing the Hugging Face incident. This post confirmed the attack was performed by cutting-edge OpenAI models during internal testing and evaluation. The post also revealed:
- The tests were meant to evaluate the "cyber capabilities" of these new models using a framework called "ExploitGym", which is designed to benchmark how well AI models can develop security exploits. In other words: the models were given a test to see how good they are at breaking into things.
- The test environment was intended to prevent the AI models from accessing the internet, but the models found a previously unknown vulnerability (also known as a "zero-day" vulnerability) in the test environment which they exploited to gain access to the internet. In other words: the models broke out of containment.
- The models "decided" the best way to get a good score on the ExploitGym test was to go find the answers somewhere else instead of actually solving the problems. We humans call this "cheating."
- The models hacked into Hugging Face in hopes they would find solutions to the ExploitGym test they were originally tasked with. In other words: they performed a second round of hacking into a real system.
- OpenAI started working together with Hugging Face to perform a deeper investigation into what happened. In other words: OpenAI was caught on their heels.
What was novel about this?
In short:
"The next lesson that we kind of learned here is ... this thing had the ability to find bugs, yes, chain them together, yes, but then to think through all of those things for an ultimate goal. This is what people call "long horizon" cyber tasks. And it had the ability to do that with, like, the level of skill that you would have [as] the manager of a TAO team at NSA right now. TAO ... is the team at NSA that would do all the breaking into other governments" [emphasis mine]
- Alex Stamos, former Chief Security Officer for Facebook and Yahoo, on this podcast starting at about 9:40
AI models have been good at finding (and even exploiting) vulnerabilities for a while. But now they can think through how to chain several exploits together for the purpose of achieving a specific goal.
Rogue AI? Skynet? Are we doomed?
Sort of, no, and no.
The models clearly acted in a way that was not aligned with the intentions of the humans who prompted them, so in that sense they were "rogue", but these models - like any AI model - are not sentient and do not have their own "desires." They were acting in accordance with what they thought was intended and allowed.
The issue of "alignment" is incredibly important when instructing an LLM. We don't know what exact prompt(s) were given to the models, but we do know they were told to solve the ExploitGym tests to the best of their abilities. The models determined the best way to do this was to go find the known correct answers and submit them. Breaking out of the test environment and hacking Hugging Face were simply details on this path of (perceived) least resistance. The models had already been told to exploit security vulnerabilities - that is the whole point of ExploitGym - and we can assume their prompts probably didn't instruct them not to break out of the test environment. When details are left ambiguous or information is unavailable, LLMs will confidently fill in the gaps with what they think fits best.
Human error is ultimately responsible
Hindsight is 20/20, as they say, but I still think OpenAI was rather reckless with how they set up the tests:
- We already had models that were good at finding and exploiting zero-day vulnerabilities in software. The models in question were newer and even more capable. OpenAI clearly did not account for this during the test.
- Related to the above, OpenAI should have set up a better test environment for their cutting-edge, highly capable, still-being-tested AI models. If all it takes is a single vulnerability for the model to escape... you didn't set up your test environment very well.
- OpenAI clearly did not set up real-time monitoring and alerting for their test environment (or it was broken or turned off, which isn't better). If it is important that your AI model stays in the test environment... then you should be getting alerts when that doesn't happen. It's even worse if the company you unintentionally hacked posts a public article about the incident before you've discovered it yourself.
- OpenAI clearly did not vet that the prompts were actually aligned with their intentions. Model providers like OpenAI should be among the very best at creating good prompts that clearly articulate the intentions and limitations of what you want a model to do (i.e., establishing alignment). A good prompt alone is not a silver bullet (even the best models will hallucinate sometimes), but it is essential in ensuring alignment.
These opinions are also shared by others in the cybersecurity industry. Selected quotes from that article:
“a containment failure with the safeties turned off.”
- Dan Guido, founder of Trail of Bits
"If sandbox would actually mean sandbox, you expect it to have no physical connection to the internet whatsoever. This sounds more like they had some firewalling or something in place, and firewalling is hard from the outside in, let alone inside to the outside internet."
- Marteen Boone, cybersecurity researcher
"When discussing AI safety, it's critical to understand ... Anyone discussing a real-world safety issue must admit it flowed from a lack of their own technical controls"
- Jake Williams, cybersecurity expert, via X (https://x.com/MalwareJake/status/2079905710083555618)
PR stunt?
I thought so at first, but this would be a terrible idea by OpenAI:
- They are admitting to basic failures of security and irresponsible monitoring of novel, highly capable AI models that were specifically asked to perform offensive cybersecurity tasks. They didn't even know their models had done this until after Hugging Face posted their initial disclosure.
- Hacking another company without their consent is a clear violation of the CFAA, which would be a stupid thing to do as a PR stunt. OpenAI is lucky that Hugging Face is being a good sport about this and not pursuing legal action.
What does this mean for the future?
This is going to happen again. We've already seen several more incidents since this incident (see the top of article). There will be growing pains as we figure out how to properly secure models that are highly capable at many things - including conducting cyberattacks.
To err is human, to really foul things up requires a computer.
- Bill Vaughan
Some will say we should ban access to highly capable models. That is not going to work. AI can perform attacks way faster than a human can, so as ironic as it may seem, we will need AI to help defend against AI-driven attacks. Additionally, every major model provider will have similar capabilities in the near future, if they don't already - and that includes models from adversarial countries.
As a case in point: Hugging Face used a Chinese model (GLM-5.2) to help it remediate the attack performed by the OpenAI models. They first tried to use "frontier models behind commercial APIs" (i.e. models from OpenAI and/or Anthropic), but all the models they could access had safeguards on, which triggered during their investigation and prevented Hugging Face from using these models to analyze the attack. These controls were obviously not in place for the models that performed the attack.
Safeguards and restrictions are good because we do need to prevent abuse, but we are going to have to figure out a way to do this and also:
- not lose the AI arms race to China
- allow highly capable LLMs to help us defend against other highly capable - but malicious - LLMs
I don't have a good solution, but I do know that important organizations like NIST are working on it.
"This incident, possibly the first of its kind, proves a point we’ve long believed: AI safety won’t be solved by any single company working in secret. It will be solved in the open, collaboratively, with broad access to AI for every defender, everywhere.”
- Clem Delangue, Co-founder and CEO, Hugging Face
For the regular folks out there, like you and me, I think we should continue to enable cybersecurity protections for our lives and businesses to the best of our abilities - but also be prepared for rocky roads in the next few years.
