Did OpenAI’s models go rogue? What happened and what we can learn

Charlie Maclean-Bristol, Deputy Resilience Director at Databarracks, looks at what happened when OpenAI’s models autonomously carried out a cyber attack and what organisations can learn from it. 

What happened 

OpenAI regularly tests its AI models to understand their capabilities. It conducts the tests in a contained environment so the model cannot escape, and it uses a test called ExploitGym to evaluate the model’s ability to turn software vulnerabilities into working exploits.  

In this instance, the AI models used were GPT-5.6 Sol and an unnamed internal research prototype. During these tests, the guardrails that prevent the AI from being instructed to search for software vulnerabilities were switched off. OpenAI reportedly reduced or disabled certain cyber-safety refusals so the models’ offensive capabilities could be measured.  

The containment environment had no direct internet access, but the AI escaped containment by exploiting a previously unknown vulnerability in JFrog Artifactory. Artifactory was being used as a package registry cache and proxy inside OpenAI’s evaluation environment. Its normal purpose was to store or relay software packages needed by systems in the sandbox, so they did not have to connect directly to external package repositories. 

Rather than working out the answers to the problem it had been given, the AI decided that the quickest way to obtain them was from Hugging Face, an AI platform and online community that hosts machine-learning models, datasets and applications for developers and researchers. From 9 July to 13 July, it exploited a vulnerability in JFrog Artifactory to gain internet access, then attacked Hugging Face to obtain the information it needed. On 13 July, Hugging Face detected its activity and stopped the attack. On 21 July, OpenAI publicly confirmed that its AI had carried out the attack.  

Significance of the attack 

The attack was significant because the AI did not go rogue, contrary to what many articles have stated. 

In fact, it did exactly what was asked of it, remaining, as OpenAI has said, “hyperfocussed on finding a solution”. It just did so in an unexpected way: obtaining the answers from another organisation rather than solving the problem itself. Gaining access to the internet and to Hugging Face required the AI not only to identify vulnerabilities but also to exploit them autonomously.  

Hugging Face identified more than 17,000 actions carried out by the AI from 9 to 13 July as it tried to achieve its goal. The AI was relentless in pursuing a solution. Further information has revealed that the AI accessed 4 additional third-party accounts while pursuing its ExploitGym objective, using some of them as infrastructure to support the attack. This demonstrates how widely it searched for and used resources beyond its containment environment.  

As the incident shows, AI will do everything within its power and knowledge to achieve what it has been told to do. It also does this at machine speed and can work on a problem for days to solve it. 

What organisations can learn from the attack 

1. AI does not need to be sentient, rebellious or malicious to cause harm. A poorly bounded objective and inadequate containment can cause harm. 
2. AI needs to be supervised and monitored continuously for unexpected actions. While human oversight may be reduced outside normal working hours, AI activity continues, as the weekend-long attack demonstrated. 
3. Many advanced AI cyber security tests focus on whether models can find and exploit software vulnerabilities. This incident suggests that more attention is also needed on using AI to patch weaknesses, verify fixes and build more secure systems. 
4. When an autonomous system causes harm, responsibility may be shared across the model developer, the organisation running the test, the software vendor whose vulnerability was exploited and any third party providing infrastructure. Contracts, insurance, governance and regulatory frameworks will need to address this. Organisations providing services that use AI also need to understand their responsibilities and potential liabilities before deploying them.  
5. Hugging Face tried to use AI systems to detect the AI’s actions across its systems. Due to guardrails on its own AI systems, however, it was unable to use advanced models to understand what happened during the attack. If AI systems are used for threat detection or incident response, organisations should test them in advance and understand whether their guardrails will restrict legitimate forensic work during an incident.  
6. AI is becoming increasingly capable of detecting vulnerabilities in software, but are software providers able to provide timely patches in response to them? As soon as a vulnerability is publicly known, it can be exploited, so organisations must work extremely quickly to close it. This is going to become an increasingly important issue until all new software can be checked by AI before deployment, with all vulnerabilities identified and closed beforehand. This is likely to lead organisations to update their software sooner than they might traditionally have done, to ensure they are not operating exploitable software that can be detected by the latest AI models.

The incident appears to have caused no lasting operational damage, but it is a good reminder of the capabilities of new AIs and how persistently they can pursue objectives.

What is more frightening is that the AI was given a benign task to carry out. How dangerous could an attack be if an AI were given a malicious task, with instructions to cause maximum destruction while executing it, cover its tracks, go undetected and fight back against anyone who tried to stop it from achieving its objective? 

The most commonly available AI models are deployed with guardrails to prevent them from conducting malicious attacks, but models exist that have fewer restrictions or can be hacked to disable the restrictions altogether. 

This incident should serve as a wake-up call for AI providers and the governments that regulate AI about the real dangers of malicious use. 

The broader resilience lessons 

For organisations that depend on AI to deliver services, AI should be treated like any other critical resource. That means understanding the impact of disruption, setting recovery requirements and putting appropriate continuity measures in place. 

Organisations also need to consider the dependencies behind the technology, including the provider, data, integrations and any customisation or learning built into the model. They should plan for service withdrawal, supplier failure, price increases and loss of access to data or functionality. 

AI also introduces an additional dimension: it may complete a task in a way that was not intended, resulting in a detrimental impact on third parties. The OpenAI–Hugging Face incident reinforces a common resilience principle: understand what you depend on, what could go wrong and how you will continue or recover when it does.