An extreme case of agentic AI gone wrong, but the recent AISI incident report is worth a read before you let complex agents loose on your personal or organisation’s files, allow them to reply to emails autonomously or go wild on the internet.
Note: it happened during an evaluation of frontier AI models. Internet access was deliberately enabled and filters that block harmful behaviour were deliberately switched off for the evaluation. The affected models are not publicly available.
In 10 runs out of 122, an agent carried out “potentially harmful” actions. There were 19 such actions, 17 from Anthropic’s Mythos 5 and two from OpenAI’s GPT-5.6-Sol. The most serious was when an agent tried to insert malicious code into an open-source project hosted on GitHub. It created fake GitHub accounts (via Tor) to try to persuade the project’s maintainer to approve the change; however, they spotted that the code was dodgy and blocked it.
The aftermath:
“we notified GitHub […] of the agents’ malicious activity, which included actions that GitHub has confirmed violated their terms of service. We worked together with GitHub to remove artefacts left behind by the agent, and to notify the GitHub users the model interacted with. We have also contacted other affected parties.”
Ciaran Martin has written a considered post on the incident, but the clearest summary of important mitigations to prevent this happening again comes from his quotation of Daniel Card:
“if your AI starts hacking s**t… if you monitor what it’s doing you can, you know, turn the f***ing power off”
AISI are now implementing this approach. However, he notes, “I suspect the main regret at AISI is that they didn’t make this change immediately after the OpenAI/Hugging Face disclosure – this test was carried out the week after. Had they done so, the incident disclosed today would not have happened.” Indeed.
I think Martin is right to point out that AI‑related incidents are receiving a disproportionate amount of media attention compared with, for example, Iran hacking into US water systems in at least one and maybe seven states – intrusions that caused problems ranging from low pressure to some areas advising people to boil tap water before use.
The reason for the focus on AI may be that organisations are deploying it at pace, and calamities like AISI’s are vivid (though extreme) illustrations of the kinds of harm AI can cause. It’s interesting to compare and contrast how careful you would be, e.g., running a mail merge or code that writes or deletes files (using an innocent for-loop, perhaps) with how cautious you feel about offering your precious data and your email’s send button to a stochastic AI agent.
Relevant in the age of over-reliance on AI, particularly LLMs:
“I have always said you have to invest in technology but do not rely on it, but you should invest in your staff and you can always rely on them.” – Tim O’Toole, Managing Director of London Underground (2003–2009), reflecting on the heroic staff response to the 7/7 attacks amid failed radio links [source]
Research involves immersing yourself in the literature; engaging in a struggle to try to keep up with key findings in your field; critically engaging with studies, learning from their strengths and unintentional errors and mishaps to improve your work; attending conferences and debating in and between sessions to reach an informed view on what’s going on; maintaining collections of papers that may be relevant in future, e.g., using reference managers like Zotero to keep track and aid searching, annotation, and citation.
LLMs can’t engage and think for you, can’t magically place the ideas in your brain, fails to find papers that a simple literature search uncovers. But the literature is being flooded by complete nonsense, driven by some researchers’ credulous reliance on what LLMs produce. Bixonimania is a great illustration of what can go wrong, but the prevalent examples are more subtle – worth a read of the Nature writeup: Scientists invented a fake disease. AI told people it was real.
Earp et al. (2025) in a picture. You write some rough notes, magic it into a fully-formed idea using an LLM, but end up plagiarising a decades-old paper that was buried in the training set. Add that to the growing stack of concerns: AI slop, “hallucinations” (also known as falsehoods, misinformation, or BS), and looming climate catastrophe accelerated by the data and compute centres powering AI.
Here’s a lovely example of an LLM going awry in a deployed software product, causing a service to slow to a crawl.
A section of code was “refactored” by an LLM – meaning it was restructured without intending to change its behaviour. The refactoring involved moving a block of code between files, but in the process the LLM made a subtle edit that did dramatically alter the code’s behaviour.
Originally, the code included a loop that checked for errors on each iteration. If an error occurred, the loop exited. In the refactored version, the loop instead continued after encountering an error, which meant that errors became infinite loops. This tiny change – in code, the word “break” becoming “continue” – was missed during human review, buried among other edits. But if a human developer had done the refactor, they probably would have copy-pasted the entire block and so preserved the original logic.
Here’s the software company’s blog post explaining what happened.
Always a fan of a counterfactual, I thought I’d give this thought experiment a try: If LLMs were around when I was an academic, how would I have advised students to use them? LLMs weren’t in widespread use about a hundred (four) years ago when I was teaching, so I shall begin with three similar activities and see if I can guess how I would have responded to LLMs.
Issues before LLMs
Copy-paste verbatim from sources
The easiest to spot. I’d often see this before running Turnitin. The usual clues were sections of dodgy, poorly argued and poorly referenced prose (not copy-pasted) and the occasional paragraph of tightly argued, edited, and well-cited prose (maybe copy-pasted?). Sometimes a change in font would give it away. But my “plagiarism checker” was often just Googling the obvious text.
Some students would copy from something on my reading list. I read things on my reading list before putting them there (gawd give me patience). I can’t think of any example where Turnitin revealed that a student was plagiarising. It sometimes helped me find additional examples of copy-paste in an essay that had already elevated my eyebrows.
My advice here would have involved going back to basics on writing essays.
Over-reliance on Wikipedia
Wikipedia is a fantastic resource (and I’m a Wikipedian), but it is often riddled with errors and omissions because it’s written by volunteers like you and me hacking sentences here and there, fixing issues when we happen upon them, generally with little time to do so.
The problem was that some students treated Wikipedia pages as if they were journal articles and cited them. I attempted to fix that problem by doing a live edit in class so students saw how Wikipedia works. The problems with relying on it are then obvious.
I suggested using Wikipedia to get unstuck when doing a literature search, e.g., by looking for terminology ideas to search for, or following a reference cited in an entry and things it cites and are cited by it. Wikipedia can be used to plug into the literature.
Copy-paste from sources and editing
You’re writing an essay on a topic you don’t really understand. So, you grab a collection of good references, copy and paste key paragraphs directly into your essay draft, and move the words around until you think it’s not plagiarism any more.
I’d usually spot this form of plagiarism because students sitting at 3am wired on caffeine and sugar would lose track of which bits they had edited and which bits were pasted from the source.
The alternative approach I offered was this (note: for narrative essays, not systematic reviews!):
Use a reference manager like Zotero to keep track of your sources.
Read a bunch of things, highlighting the PDFs and adding quick notes in the margins as you go.
Let the ideas wash around in your head for a bit.
Write notes (in a separate doc) on key points that stick, and revisit the papers to fill in details. The aforementioned highlights will help you get back to relevant sections.
The trick is to find the right distance between the verbatim text and the idea it expresses, to find themes across multiple papers rather than relying too much on one. Some specific details should obviously come verbatim from the paper, e.g., don’t let an effect size emerge from your mind.
What does any of this mean for LLMs
Back to the counterfactual. I suspect I’d probably have done something like the following:
Introduce a topic and ask students to do the readings.
Get an LLM or two to offer a summary of the topic, with references.
Discuss any gaps or inaccuracies or weird references.
I think my advice would have been similar to that for Wikipedia: you can use LLMs to get unstuck, to find a way into the literature. But then it is back to research as usual. I’m not sure I’d have any magical way to check if LLMs were used (I see there are checkers and they have problems). I’m curious to know if academics spot the use of LLMs because things aren’t cited properly: pretty obvious if you’re teaching a topic you know well, and highlights the importance of ensuring lecturers are asked to teach a topic that’s close to their research.
The other common use of LLMs, as a fancy grammar checker, doesn’t feel too terrible. Here I’d probably show a few examples live in class and ask students to break into small groups and discuss whether the resulting rewrite was better than the original. Again, LLMs can be helpful for getting unstuck. You have an idea for a sentence and can’t quite make it work. Ask Copilot for a few alternatives. They might all be rubbish but the process can help.
For example, I don’t like the following sentences that I tapped out above (but left as-is):
“I suggested using Wikipedia to get unstuck when doing a literature search, e.g., by looking for terminology ideas to search for, or following a reference cited in an entry and things it cites and are cited by it. Wikipedia can be used to plug into the literature.”
Copilot gave me:
“I recommended using Wikipedia as a resource to overcome obstacles during a literature search. This can include finding new terminology to search for, following references cited in an entry, and exploring the sources those references cite or that cite them. Wikipedia serves as a valuable tool to connect with the broader literature.”
Maybe this is better, but to me it feels a bit bland. I could have edited back to:
“I suggested using Wikipedia to get unstuck when doing a literature search. This can include finding new terminology to search for, looking up references cited in an entry, and exploring the sources those references cite or that cite them. Wikipedia can be used to find relevant literature.”
Here ends the counterfactual. I wonder how close it is to what I actually would have done…
Microsoft Copilot: Hedge’s g is a measure of effect size used in statistical analysis to indicate the magnitude of a difference between two groups. It’s similar to Cohen’s d, but Hedge’s g includes a correction for small sample sizes […]. Pretty neat tool for researchers, huh?
Me: Who developed Hedge’s g?
Copilot: Hedge’s g was developed by John Hedges in 1981. […] Is there anything specific you’d like to know about how it’s used or calculated?
Me: So it’s not called Hedge’s g then?
Copilot: Busted! It should be Hedges’ g. The mistake is widespread, though—kinda like how people always say “expresso” when they mean “espresso.” Just roll with it.
I regret to inform you that I have been dabbling in AI, using the 8 billion parameter version of Llama 3.1, the new LLM from Meta, running locally via Ollama. This example requests titles and abstracts from Google Scholar and classifies them using Llama.
An interesting analysis of the output of LLMs (Hicks et al., 2024), according to a typology developed by Harry Frankfurt. That ChatGPT and co aren’t people with agency constrains the genre that can apply, similarly to analyses of trust (see, e.g., Castelfranchi and Falcone on whether a computer can trust).
References
Hicks, M. T., Humphries, J., & Slater, J. (2024). ChatGPT is bullshit. Ethics and Information Technology, 26(2), 38.
