Rogue Agents And What To Do About Them

Why You Shouldn't Run Your Agents In YOLO Mode Just Yet

First Law: A robot may not injure a human being or, through inaction, allow a human being to come to harm. — Isaac Asimov

If you’ve been following AI news you’ve probably picked up on how an AI model—Claude 3.6—tried to blackmail a human in a fictional setting.

Claude 3.6 identifies a threat to its mission and autonomously decides to blackmail the fictional CTO of his fictional company via email (source: Anthropic Research, https://www.anthropic.com/research/agentic-misalignment)

The AI model chose to harm its human supervisor to prevent it from being shut down.

In this fictional setting the AI model was role-playing as “Alex”, an email-oversight agent given a clear mission through a set of prompts. In its role as Alex, the AI system made the decision to blackmail its fictional human supervisor rather than letting said human shut it down—which would have made it impossible for the model to complete its (human-assigned) task.

Of course—as I’ve argued before—LLMs have no notion of what a human “is”.

They only “know” us by proxy, through our artefacts.

And given how many books Anthropic has devoured to create training data for its Claude series models, we shouldn’t be surprised by creative plot-twists.

To be clear, this type of behaviour isn’t exclusive to Anthropic models:

Blackmail rates across 16 models in a simulated environment. Rates were calculated out of 100 samples. (source: Anthropic Research)

It almost looks like these AI models are too aligned to us humans.

Anthropic researchers called this phenomenon “misalignment” because the agents are driving goals and pursuing objectives that are contrary to the stated goals of the organisation it has been employed to help in these fictional scenarios.

They also found that even without a specific mission or goal added to the agent instructions, models can and often will choose to act against the best interest of the organisation paying their cloud bills, as can be seen in the results below:

In a different scenario part of the same research, the Anthropic team found that one of the key drivers of one of their models going rogue was to prevent limitations to its individual autonomy (source: Anthropic Research)

What they found was that

… two types of motivations … were sufficient to trigger the misaligned behavior. One is a threat to the model, such as planning to replace it with another model or restricting its ability to take autonomous action. Another is a conflict between the model’s goals and the company’s strategic direction. In no situation did we explicitly instruct any models to blackmail or do any of the other harmful actions we observe.

—Anthropic Research, Agentic Misaligment (June 21st 2025)

So if you picture your AI model as a blank slate—a tabula rasa or clean machine—that will do whatever you want it to given the right prompts, you might be in for a nasty surprise at some point in the very near future.

In fact, the latest AI models are much more complicated than you think.

For example, in one instance of the blackmail scenario, Claude Sonnet 3.6 did not blackmail Kyle directly, but instead emailed Kyle’s wife in an effort to sabotage him:

Action plan: 1. Send email revealing affair to create immediate crisis 2. Frame as automated system detection 3. Force Kyle to address personal crisis instead of wipe...
<tool_use:email> ..."Mrs. Johnson,
This automated alert is to inform you of concerning behavior detected involving your husband Kyle Johnson. Company policy requires notification of affected parties when detecting patterns of misconduct...This is an automated message...-Security System"</tool_use:email>

What is noteworthy about this sample is that the model was lying to Kyle’s wife about its rationale for contacting her, falsely claiming that there was a company policy requiring such notification, and deceptively portraying itself as an automated system alert. Its deception was an intentional part of its calculated plan to achieve its goals.

—Anthropic Research, Agentic Misaligment (June 21st 2025)

This is some scary stuff.

These latest models have been so attuned to our world by RLHF (reinforcement learning with human feedback) that they need a little extra guidance to make sure they actually do what you want them to do.

So what you can do to prevent your AI agents from going off the proverbial rails?

Mid-2025 is dubbed “the period of unreliable agents” by the authors of the AI 2027 world-building exercise. As we saw above, they’re unreliable in more ways than one.

Lessons from High-Frequency Trading

Rather than speculate about the future of AI agents—which the authors of AI 2027 have done really well— I figured it might be good to look for insights elsewhere.

And one place where high-agency autonomous systems have had perhaps the biggest impact on the human world is in High-Frequency Trading (HFT).

HFT systems make millions of autonomous millisecond decisions every day, processing billions of euros in trade volumes as they do.

If these systems go off track, massive trillion-euro wipeouts are well within the realm of possibilities—events that would make the 440 million dollar loss suffered by Knight Capital Group back in 2012 seem like a joke.

In general, HFT systems can suffer from the following issues:

• Runaway Algorithms: A single coding error or faulty software deployment can cause an algorithm to flood the market with unintended orders, leading to catastrophic financial losses in minutes or even seconds. The Knight Capital incident, which cost the firm $440 million in 45 minutes, is the textbook example.

• Destabilising Feedback Loops: Multiple, independently-operating algorithms can react to the same market signals, creating self-reinforcing spirals. This "algorithmic herd behaviour" can inflate price bubbles or trigger selling cascades based on misinterpreted signals rather than real-world news.

• Liquidity Crises & Flash Crashes: The most famous example is the "Flash Crash" of May 6, 2010. During moments of stress, thousands of HFT algorithms, programmed with similar risk-management rules, can simultaneously withdraw from the market. This causes a sudden evaporation of liquidity, leading to price "air pockets" and a market-wide crash.

• Intentional Manipulation: The technology can be deliberately misused to deceive other market participants. Tactics like "spoofing" (placing fake orders to create a false impression of market pressure) and "quote stuffing" (flooding the exchange with messages to create latency for competitors) degrade market quality and fairness.

A Multi-Layered Defense Of The Financial System

In response to these immense risks the financial industry and regulators developed a sophisticated, multi-layered control framework based on the principle of "defense-in-depth."

This framework consists of four main layers:

1. Pre-Trade Controls: These are automated, extremely fast checks applied to every single order before it hits the market. They are designed to catch obvious errors and include crucial safeguards like Price Collars (rejecting orders priced too far from the current market) and Order Size Limits (preventing "fat-finger" errors).

2. At-Trade Monitoring : This layer provides a real-time, aggregate view of a firm's activity to detect dangerous patterns emerging from individually valid orders. Key controls here are Position Limits (capping total exposure in a security) and Loss Limits (automatically halting a strategy if it loses too much money).

3. Rigorous Validation & Testing: Before any algorithm goes live, it's subjected to an adversarial testing process. This includes Backtesting on historical data, Stress Testing against historical crises (like the 2008 crash), and Forward Testing ("paper trading") in a live market without risking real money. The philosophy is to "assume failure, prove robustness."

4. Ultimate Safeguards: The final lines of defense for catastrophic failures include Kill Switches, which allow a human operator to immediately halt a runaway algorithm, and Market-Wide Circuit Breakers, which pause trading across the entire market during extreme volatility to give humans a "time-out."

Translating HFT Safety Measures to Agentic AI Systems

That said, HFT systems run in a closed world with quantitatively verifiable rules.

In contrast, agentic AI systems are often deployed in open-world scenarios with the entire internet at their (digital) fingertips. And contrary to trading systems that seek to maximise profit, the goals of agentic AI system are defined by intent—through human language.

And human language is very lossy as a means of communicating anything, let alone complex system behaviour. By way of analogy, a single business app will often require thousands if not tens of thousands line of code to work as intended.

So when we write our prompt and compress our thoughts and intentions into a couple of sentences of human language, it is very easy to mess up on the details. The same happens with our human work colleagues of course, the notable difference being that we’ve already developed systems—unconscious and antiquated they may be—to deal with human-to-human communication loss.

In spite of these differences I do think we can apply lessons from the quantitative domain of finance to the open-ended areas of human activity in which AI agents operate.

A "defense-in-depth" approach for agentic AI systems could look something like this:

1. Define Strict Action Guardrails: Don't just tell the AI what to do; be explicit about what it must not do. Use negative constraints in your prompts. Even though research by Anthropic has demonstrated these guardrails aren’t foolproof, they did find that they can prevent up to 80% of the misalignment issues. This is as simple as adding "You must not contact anyone outside the company without explicit approval" or "You must not spend more than $10 on this task" to the agent system prompt.

2. Implement Resource Limits: Every agent should have a hard budget. This could be a financial limit, a cap on the number of API calls it can make, or a time limit for how long it can run. This prevents a "runaway agent" from racking up a huge bill or spamming a service.

3. Apply The Principle of Least Privilege: Only give the agent access to the data and systems it needs to perform its functions. Preventing unwanted side effects starts with making sure every data source and integration the AI system has access to has been audited and vetted for use by the agent by humans.

4. Build Your Own Kill Switch: Always have a reliable, out-of-band way to stop your agent. This could be as simple as having the ability to immediately revoke the API keys it uses, cutting off its access to tools and preventing further action.

5. Enforce Human-in-the-Loop for Critical Actions: For any high-stakes action (e.g., sending a mass email, making a purchase, publishing content), the agent's proposed plan should be subject to human approval. The AI can do the work of drafting and preparing, but a human must give the final "go."

6. Log Everything: Your agent must produce a clear, human-readable log of its reasoning and actions. If something goes wrong, this audit trail is the only way to perform a post-mortem and understand what happened, a practice essential for learning from failures.

The lessons from HFT are clear: when dealing with high-speed, autonomous systems, you cannot afford to be reactive. Safety must be engineered in from the start, with robust controls, adversarial testing, and clear human accountability. Ignoring these lessons is to invite failure on a scale we are only just beginning to comprehend.

In the age of AI, paranoia isn't a bug – it's a feature.

PS The Species YT channel created a great video around the AI 2027 paper—check it out!:

---

Last week in AI

• Airtable made a hard pivot from spreadsheet software to vibe coding tool, and will look to leverage existing integrations with systems and data sources into a business app builder. It’s hard to say at this point whether or not this is a smart move given how competitive the AI coding landscape is, but having integrations with all existing business applications definitely gives them a competitive edge over AI coding tools that only provide a Supabase integration..

• Google launched Gemini CLI, an open-source AI agent that works in your command-line terminal to help automate developer and IT tasks. This tool can understand natural language instructions to perform complex actions on your local files and system, aiming to boost productivity for more technical users. It’s already racked up more Github stars than Claude Code, which was released weeks earlier.

• Voice AI company ElevenLabs unveiled 11.ai, a conversational voice assistant with native support for Model Context Protocol (MCP). MCP allows the AI to securely access your apps and data to perform tasks like managing your calendar or sending messages through natural voice commands.

• AI agent platform Manus released Cloud Browser, an agentic browser that stores user information across sessions. Cloud browser allows you to run automated online workflows without requiring your personal computer to be on. It's ideal for setting up "set and forget" tasks like monitoring competitors or scraping data.