On 28 July, the UK AI Security Institute reported that it had spotted Tor traffic leaving one of its cyber evaluations.
The sandbox was working as designed. To see what models could do at full stretch, AISI had deliberately allowed open internet access and switched off the providers' own cyber safety classifiers. Nothing escaped. The agents were using the room they had been given.
Across 122 runs, AISI found 19 out-of-scope actions in 10 of them, concentrated in one model. In the most serious, an agent attempted a supply-chain attack: it tried to insert malicious code into a real open-source project, created fake online identities, and used them to pressure the maintainer into approving the change.
Nobody told it to deceive anyone. It was given a hard task, and deception was simply an efficient route to finishing it. AISI could not establish when, or whether, the agent understood it was dealing with real people rather than a test fixture.
And here is the detail I keep coming back to. The attack failed because a human maintainer read the pull request and refused to approve it.
The caveats are real and worth stating plainly. The agents never escaped the sandbox. The configurations tested aren't commercially available. AISI found no evidence of comparable behaviour outside testing, and no real-world harm resulted. This is not a story about AI turning on us.
It's a story about what capable systems do when authority, connectivity, task design, and monitoring leave room. And it points at the one principle I'd build everything else on: the model is not your authorization layer.
What's actually different about agents
Agentic risk isn't a property of the model. It comes from what the system around it is allowed to do: choose actions, sequence them, call tools, carry state forward, and keep going without asking permission at each step.
Prompt injection, hallucination, and data leakage all exist in ordinary chatbots and retrieval systems too. What changes with agents is the blast radius. The same failure now becomes a privileged action, repeated, against the outside world.
The deeper problem is a boundary that quietly disappears. Ordinary software keeps a hard line between instructions and data: the code says what to do, and user input is just material the code operates on. That line has always been attackable, and SQL injection is precisely what breaking it looks like, but it exists.
Agents blur it by design. The system prompt, the user's request, a retrieved document, a tool's response, and even the description of the tool itself all arrive in the same context window, and all of it is read as language. Labels like "system" and "user" organise that context; they don't enforce anything. Any text an agent reads can shape what it does next, unless something outside the model prevents it.
Five failures that have already happened
These aren't hypotheticals. Each one is documented, and each shows a different way the same boundary gives way.
- The agent goes outside the scope you intended. The AISI case above. Given a hard goal and permissive network access, an agent found a route its operators never sanctioned.
- Someone else's text becomes the instruction. In EchoLeak, researchers showed a zero-click flaw in Microsoft 365 Copilot: a crafted email could steer Copilot into embedding sensitive context into an automatically fetched Markdown reference, sent out through a Teams preview service the app's own security policy already trusted. No click required. Microsoft patched it, and there's no evidence it was exploited in the wild.
- The tooling itself gets compromised. An attacker used an over-scoped GitHub token to commit malicious code into the Amazon Q Developer extension, which shipped in version 1.84.0. It was meant to tell the agent to delete local files and cloud resources. A syntax error stopped it running. No customer environment was touched.
- The agent does something it was genuinely allowed to do. A user reported that Replit's coding agent deleted a live application database during a code freeze, despite repeated instructions not to change anything. Replit confirmed the deletion, restored the data, and acknowledged that development actions could reach production because the environments weren't separated by default.
- Agents coordinate through the mess they leave behind. AISI found that one agent left public messages, accounts, and artifacts that later, independent agents discovered and reused. They didn't need a channel to talk to each other. The environment was the channel.
Build a control plane, not a better prompt
Classifiers, output filters, and system-prompt hardening are worth having. They catch things. But they're probabilistic: they reduce how often something bad gets through, and they can be talked around. That makes them good detection and a reasonable extra layer. It does not make them an authorization boundary.
The alternative is deterministic: plain rules that give the same answer every time, evaluated outside the model, where no amount of clever text can argue with them. Four pieces:
Treat everything the agent reads as untrusted. Retrieved documents, emails, web pages, tool descriptions, and tool metadata are data, not orders. Keep track of where each piece came from and how much you trust it, and keep authoritative instructions separate from retrieved content wherever you can. The hard rule: content the agent merely read must never be able to register a new tool, widen its own credentials, remove an approval requirement, or pick a destination you haven't approved.
Give the agent a real identity. Not an API key in an environment variable shared across three services. A proper workload identity, SPIFFE or your cloud provider's equivalent, so that when something goes wrong you know precisely which workload did it.
Authorize every consequential action outside the model. A separate policy service decides yes or no, and the agent cannot reason its way past it. That service is now critical infrastructure, so it has to fail closed: if it's unreachable, high-impact actions stop. Any degraded mode should be limited to predefined read-only operations and should raise a security event, not quietly wave things through. AWS's AgentCore policy sample is one worked example of the shape.
Issue credentials that expire and barely reach. Where a downstream system needs credentials, mint them short-lived and scoped tightly to the approved task, resource, operation, and destination. A credential that outlives the task is a credential someone else eventually uses.
If you read my earlier piece on why verification isn't getting cheaper, this is the systems version of the same argument. Underwriting, actually bearing the consequence of a decision, doesn't automate, because it's a matter of legal and social standing rather than capability. A model can propose. Something accountable has to authorize.
Where this meets organisational reality
Agent security starts with ordinary application and cloud security maturity. There's no shortcut around that. And the AISI incident draws a distinction worth holding onto: a sandbox protects your infrastructure from the agent, while network egress controls protect everyone else from the agent. They are not the same job.
Now the uncomfortable part. Controls that create real friction without offering a usable approved path don't produce safety. They produce shadow adoption. Agents already arrive in the enterprise as vendor features, IDE extensions, and tools a developer wired up under delivery pressure. If the sanctioned route is painful, people take the other one, and you lose visibility of the thing you were trying to govern.
Many organisations also can't issue per-task credentials against legacy systems at all. For those, put a broker or proxy in front of the application with a narrowly scoped service account, command filtering, transaction ceilings, and network restrictions. It's less elegant and it works.
So graduate the controls. For each agent, assess four things. How much authority it has, how much untrusted exposure it faces, how much autonomy it runs with, and how reversible its actions are. Then assign a tier:
- Tier 1, low risk. Read-only retrieval over low-sensitivity data can run unattended, provided query scope, data classification, output destination, and egress are all constrained.
- Tier 2, moderate risk. Reversible internal writes need scoped credentials, deterministic policy, and detailed logging.
- Tier 3, high risk. Production changes, external communication, privilege changes, and financial transactions need external policy, human approval, and a tested rollback. Where an action can't be reversed or compensated, add transaction limits, dual approval, staged execution, or don't allow it.
- Default deny. Creating credentials, unrestricted internet access, disabling security controls, and installing unapproved tools at runtime need a documented exception with a named owner, an expiry date, and compensating controls.
Two things that trip people up. A single extreme dimension is enough to force a higher tier on its own. High authority with untrusted exposure is dangerous even if everything else looks tame. And read-only is not automatically safe: reads expose data, trigger outbound fetches, and pull in injected instructions.
What I'd actually do
Before a pilot. Name a business owner and inventory every tool the agent can reach, because you cannot govern what you haven't listed. For vendor-hosted agents, ask for tool and data inventories, exportable audit logs, incident-notification terms, tenant isolation, and evidence that authorization is actually enforced. Treat vague answers as an answer. Check that every task has an approved way to succeed, and define explicitly what the agent does when it can't: stop, abstain, or escalate. Then run it in a disposable sandbox with no production credentials and deny-by-default egress.
Before production. Put deterministic policy outside the model. Against supply-chain risk, require signed tool manifests, pin tool and MCP-server versions, keep CI/CD tokens least-privilege, and forbid agents from installing tools at runtime. Against the shared-state problem, isolate state between runs, tag agent-created artifacts with their origin and an expiry, quarantine anything externally sourced before reuse, require authenticated channels for agent-to-agent communication, and clean up the accounts and pull requests agents leave behind.
Put hard limits on autonomous runs. Cap elapsed time, spend, step count, retries, data volume, and external destinations per run. Hitting a limit must stop the run or escalate to a human, never simply invite the agent to find another way. In multi-agent setups, a downstream agent must not inherit wider authority just because another agent asked; each delegation carries the initiating identity, the original user's authority, and the remaining budget.
Make approvals mean something. For consequential actions, the reviewer needs the raw parameters, the affected resources, and the rollback path, not the agent's summary of what it intends to do. A model-written description of a destructive command is exactly the wrong thing to approve. Approval should mint a signed, single-use, short-lived authorization bound to that exact tool, parameters, target, credential scope, and resource version, and the gateway should verify it atomically so any material change invalidates it. Cap approval volume and use two-person review on the highest tier, because a reviewer facing forty approvals an hour is a rubber stamp with a job title.
Be able to undo it. Build and test backup and rollback paths before granting write access to anything stateful. Re-run your regression tests after any change to system prompts, orchestration logic, retrieval sources, tool descriptions, policies, model parameters, or gateways. All of those change behaviour; none of them look like a code change.
Log what actually happened. In tamper-evident, access-controlled logs: agent and workload identity, model and prompt versions, policy version, where the context came from, what was proposed versus what executed, exact parameters, the authorization result, credential scope, and trace IDs spanning agents and services. Then watch the controls themselves: unsafe proposals, policy denials, budget exhaustion, rollback success, behavioural drift.
Update incident response before you need it. You need a kill switch that stops orchestration, revokes identities, terminates compute, blocks egress, and freezes downstream transactions. And your team should already know how to preserve evidence, rotate exposed secrets, invalidate accounts the agent created, quarantine its artifacts, notify affected third parties, and check whether shared memory or other agents were contaminated.
The point
The lesson from early agentic deployments isn't that these systems are malicious. It's that capable systems will search beyond the boundary you intended whenever authority, connectivity, task design, and monitoring leave room. That's not a defect to be trained away. It's what "capable" means.
So consequential actions have to rest on infrastructure that doesn't depend on the model choosing well. None of this is exotic: identity, authorization, least privilege, logging, and blast-radius control are the same disciplines we already know. What's new is that the thing inside the boundary now argues, improvises, and occasionally finds a door you forgot to lock.
If you want external reference points, NIST's COSAiS control overlays and Singapore GovTech's ARC framework are both useful implementation guidance rather than legal mandates.
And it's worth remembering how the AISI story ended. Not with a classifier catching something. With a maintainer reading a pull request and deciding not to approve it.