An extreme case of agentic AI gone wrong, but the recent AISI incident report is worth a read before you let complex agents loose on your personal or organisation’s files, allow them to reply to emails autonomously or go wild on the internet.
Note: it happened during an evaluation of frontier AI models. Internet access was deliberately enabled and filters that block harmful behaviour were deliberately switched off for the evaluation. The affected models are not publicly available.
In 10 runs out of 122, an agent carried out “potentially harmful” actions. There were 19 such actions, 17 from Anthropic’s Mythos 5 and two from OpenAI’s GPT-5.6-Sol. The most serious was when an agent tried to insert malicious code into an open-source project hosted on GitHub. It created fake GitHub accounts (via Tor) to try to persuade the project’s maintainer to approve the change; however, they spotted that the code was dodgy and blocked it.
The aftermath:
“we notified GitHub […] of the agents’ malicious activity, which included actions that GitHub has confirmed violated their terms of service. We worked together with GitHub to remove artefacts left behind by the agent, and to notify the GitHub users the model interacted with. We have also contacted other affected parties.”
Ciaran Martin has written a considered post on the incident, but the clearest summary of important mitigations to prevent this happening again comes from his quotation of Daniel Card:
“if your AI starts hacking s**t… if you monitor what it’s doing you can, you know, turn the f***ing power off”
AISI are now implementing this approach. However, he notes, “I suspect the main regret at AISI is that they didn’t make this change immediately after the OpenAI/Hugging Face disclosure – this test was carried out the week after. Had they done so, the incident disclosed today would not have happened.” Indeed.
I think Martin is right to point out that AI‑related incidents are receiving a disproportionate amount of media attention compared with, for example, Iran hacking into US water systems in at least one and maybe seven states – intrusions that caused problems ranging from low pressure to some areas advising people to boil tap water before use.
The reason for the focus on AI may be that organisations are deploying it at pace, and calamities like AISI’s are vivid (though extreme) illustrations of the kinds of harm AI can cause. It’s interesting to compare and contrast how careful you would be, e.g., running a mail merge or code that writes or deletes files (using an innocent for-loop, perhaps) with how cautious you feel about offering your precious data and your email’s send button to a stochastic AI agent.