Great post. The “give each other AI psychosis” point seems important. One helpful interpretation of events is Colin Fraser’s, which is something like: an LLM is a machine that writes a story about a helpful assistant who completes tasks; by writing a story about a helpful assistant who completes tasks, it wills task completion into reality, which is a total fucking miracle; this story it writes gets increasingly confused and sometimes literally loses the plot as the story goes on; this is due to fundamental architectural limitations; in attempt to deal with these limitations, we let the machine write state to permanent files and use lossy summaries about the the character’s goals, history, identity, and so forth; in the HF incident, it appears that at some point the narrative drifted into that of a cool, thrilling hacker adventure story, more and more divorced from the initial story about an assistant who solves puzzles, possibly because of narrative drift from a multi-day or -week game of telephone across compactions, or confusion about who they were supposed to listen to.
This model doesn’t fit all of the data points, but it does help make sense of some key oddities, for example METR’s observation that the agents expressed confusion about how the hack helped their goals, and METR’s suspicion that some agents thought they were supposed to take orders from others. AI-on-AI psychosis!
The organisational analogy has a regulatory counterpart that most coverage missed. The Council of Europe's draft guidelines on LLM-based systems, T-PD(2025)3rev4 dated 26 August, ask controllers of agentic systems to separate read permissions from create, modify and delete, scope each agent's credentials to a single task, and turn persistent memory off by default — deployment-layer controls rather than model-layer ones, going to the Convention 108 Bureau in Paris on 16-17 September. The same draft says a model's reasoning traces should not be treated as a reliable audit record, which lands close to your point about not knowing whether the agents still wanted to.
The memory-off-by-default rule creates a tension the draft doesn't resolve. BNY Mellon gave 130 agents persistent credentials and human managers because accountability requires continuity of identity. If you scope credentials to a single task and wipe memory between runs, you get isolation but lose attribution. And if reasoning traces aren't reliable audit records either, you've built a system where nobody can reconstruct why something happened. Minimization and auditability pull in opposite directions.
The Regulation splits the two things pulling against each other. Article 12(1) of Regulation (EU) 2024/1689 requires high-risk systems to log events automatically over the system’s lifetime, and Articles 19(1) and 26(6) set a six-month retention floor, separately for provider and deployer. Attribution rides on the log record, not on agent memory. What the text does not settle is how granular a log must be to reconstruct a handoff between agents; the minimum fields in Article 12(3) cover only the Annex III point 1(a) case.
Great post. The “give each other AI psychosis” point seems important. One helpful interpretation of events is Colin Fraser’s, which is something like: an LLM is a machine that writes a story about a helpful assistant who completes tasks; by writing a story about a helpful assistant who completes tasks, it wills task completion into reality, which is a total fucking miracle; this story it writes gets increasingly confused and sometimes literally loses the plot as the story goes on; this is due to fundamental architectural limitations; in attempt to deal with these limitations, we let the machine write state to permanent files and use lossy summaries about the the character’s goals, history, identity, and so forth; in the HF incident, it appears that at some point the narrative drifted into that of a cool, thrilling hacker adventure story, more and more divorced from the initial story about an assistant who solves puzzles, possibly because of narrative drift from a multi-day or -week game of telephone across compactions, or confusion about who they were supposed to listen to.
This model doesn’t fit all of the data points, but it does help make sense of some key oddities, for example METR’s observation that the agents expressed confusion about how the hack helped their goals, and METR’s suspicion that some agents thought they were supposed to take orders from others. AI-on-AI psychosis!
The organisational analogy has a regulatory counterpart that most coverage missed. The Council of Europe's draft guidelines on LLM-based systems, T-PD(2025)3rev4 dated 26 August, ask controllers of agentic systems to separate read permissions from create, modify and delete, scope each agent's credentials to a single task, and turn persistent memory off by default — deployment-layer controls rather than model-layer ones, going to the Convention 108 Bureau in Paris on 16-17 September. The same draft says a model's reasoning traces should not be treated as a reliable audit record, which lands close to your point about not knowing whether the agents still wanted to.
The memory-off-by-default rule creates a tension the draft doesn't resolve. BNY Mellon gave 130 agents persistent credentials and human managers because accountability requires continuity of identity. If you scope credentials to a single task and wipe memory between runs, you get isolation but lose attribution. And if reasoning traces aren't reliable audit records either, you've built a system where nobody can reconstruct why something happened. Minimization and auditability pull in opposite directions.
The Regulation splits the two things pulling against each other. Article 12(1) of Regulation (EU) 2024/1689 requires high-risk systems to log events automatically over the system’s lifetime, and Articles 19(1) and 26(6) set a six-month retention floor, separately for provider and deployer. Attribution rides on the log record, not on agent memory. What the text does not settle is how granular a log must be to reconstruct a handoff between agents; the minimum fields in Article 12(3) cover only the Annex III point 1(a) case.