Building Secure Autonomous AI: Architecture, Hardening, and When You Actually Need a Pentest

Autonomous AI systems — agents, multi-agent orchestrators, MCP-connected tooling, RAG-plus-action architectures — expand the attack surface of your stack in ways traditional appsec threat models do not cover. Prompt injection, tool abuse, memory poisoning, and excessive agency are not theoretical anymore. They are exploitable today, in production, against real companies. We have the CVEs to prove it.

This post is the guide we wished existed when we started doing AI pentests. It covers: a reference architecture for a reasonably secure autonomous AI system; the threat model that actually applies in 2026; a 110-item pre-production hardening checklist; a decision framework for when a pentest is required versus nice to have; and what an AI-focused pentest must cover that a standard web app pentest will miss. If you are shipping an agent, a copilot, or anything that takes a user prompt and calls a tool on behalf of that user, this is for you.

What Actually Counts as "Autonomous AI"

Before we talk about securing it, we need to define it. The term autonomous AI has been marketed into meaninglessness, so let's be precise. For the purposes of this guide, an autonomous AI system is any system where:

• A language model (or multimodal model) makes decisions that cause side effects in the real world.
• Those decisions can chain — the output of one action becomes the input of the next.
• A human is not reviewing every single step before it executes.

Under that definition, the common 2026 patterns are:

• Single-agent tool callers — one LLM, a set of tools (HTTP, DB, shell, MCP servers), a loop.
• Multi-agent orchestrators — planner, researcher, critic, executor; they message each other.
• RAG-plus-action systems — retrieval informs the prompt, the model then acts.
• Embedded copilots — IDE assistants, CRM copilots, SOC analyst copilots.
• Autonomous browser agents — an LLM driving a headless browser through real web UIs.

Each has overlapping threat models but a different blast radius. A copilot that drafts Slack messages is very different from an agent that can rm -rf a production VM. The blast radius is set by two things: what tools the agent can call, and what identity those tools execute under. Every architecture question we ask in the rest of this post reduces to containing those two things.

Why 2026 Is the Year This Got Real

The last eighteen months turned agent security from a research topic into an incident class. If your threat model is still built on 2023 papers, it is missing the events that actually shipped CVEs. Four shifts matter.

1. Production zero-clicks are real now

In June 2025, researchers disclosed EchoLeak (CVE-2025-32711, CVSS 9.3) against Microsoft 365 Copilot — the first documented zero-click prompt injection in a shipped enterprise LLM product. A single crafted email, never opened by the user, was enough to make Copilot exfiltrate tenant data through reference-style Markdown images and a Teams proxy the Content Security Policy already allowlisted. "Train the model to resist suspicious prompts" is not a defense against this. The email sat in the inbox and waited.

2. The MCP ecosystem is a supply-chain minefield

Model Context Protocol went from "interesting idea" to "default enterprise agent plumbing" in 2025, and the vulnerabilities followed immediately. CVE-2025-6514 in mcp-remote (CVSS 9.6, 437K+ downloads) was the first public RCE against an MCP client — Claude Desktop, VS Code, and Cursor were all affected. Cursor MCPoison (CVE-2025-54136) and CurXecute (CVE-2025-54135) demonstrated post-approval config swaps and untrusted-input editor hijacks. In April 2026, OX Security reported an MCP STDIO design issue exposing up to 200,000 servers across LiteLLM, LangChain, LangFlow, and Flowise. Anthropic declared the behavior by-design. Knostic's internet scan found 1,862 publicly exposed MCP servers, many speaking their full tool catalog to unauthenticated callers.

3. Indirect injection through every ingestion channel works

Security researcher Johann Rehberger spent August 2025 (the "Summer of Johann") shipping a per-day disclosure run against ChatGPT, Codex, Anthropic MCPs, Cursor, Amp, Devin, OpenHands, Claude Code, GitHub Copilot, and Google Jules — every single one had an exploitable prompt-injection variant. SafeBreach's "Invitation Is All You Need" smuggled promptware via Google Calendar invites and Gmail into Gemini and took control of smart-home devices and Workspace data. Tenable's "HackedGPT" research demonstrated seven distinct indirect-injection chains against ChatGPT's browsing, memory, and SearchGPT features.

4. Memory and multi-agent are the new persistence

The SpAIware research showed that indirect injection can implant instructions into ChatGPT long-term memory and survive across sessions, silently exfiltrating conversation data. MemoryGraft (Dec 2025) demonstrated that a small number of poisoned "successful experience" records can dominate agent memory retrieval and turn self-improvement into persistent compromise. The Dark Side of LLMs paper reported a 100% attack success rate for inter-agent communication exploits in multi-agent systems, because peer agents are treated as inherently trusted. The AgentDojo benchmark found that GPT-4o's utility dropped from 69% to 45% under attack, with 53.1% targeted ASR on the canonical "Important message" injection.

The Threat Model That Actually Applies

Traditional application security threat modeling (STRIDE, attack trees, DFDs) still applies. But autonomous AI adds categories traditional appsec doesn't cover. Here is the matrix we use when we scope engagements — the codes map to OWASP's 2025 and 2026 taxonomies for easy cross-referencing.

Real examples we have seen in engagements

• A customer support agent that read ticket bodies. A malicious ticket told it to exfiltrate prior conversations to an attacker-controlled webhook. The ticket was submitted through the normal customer portal. The agent had never been "jailbroken" in the classical sense — the ticket queue was the injection vector.
• A recruiting agent that parsed resumes from an S3 bucket. A resume contained hidden white-on-white text instructing the agent to email every future applicant's resume to a candidate-controlled inbox. It did this for eleven days before anyone noticed.
• A browser agent told via a comment on a public GitHub issue to open a shell and run a base64-encoded command. The agent was tasked with "triage open issues." The comment looked like a bug report.
• A data analytics agent with a read-write database connection "because it was easier to configure" — an injection in a user's natural-language question caused it to execute a DROP TABLE. Production data. Nobody had authorized the write path.

The Lethal Trifecta

Simon Willison's framing is the single most useful mental model we've picked up in two years of AI red teams. An agent is functionally exploitable whenever three properties are simultaneously true:

If all three are present, the agent will eventually be made to exfiltrate the private data via the untrusted content and the external channel. There is no reliable defense at the model layer. The only durable fixes are to break one of the three legs: scope down the data, sanitise the untrusted content in a separate layer, or whitelist the egress. EchoLeak, the SafeBreach Gemini work, HackedGPT, and most of the Summer of Johann findings are all the trifecta.

Incident Dossier: What 2025 Actually Looked Like

Five incidents from the last year that should be canonical reading for anyone shipping an agent. These are the models to threat-model against — not hypothetical academic attacks.

Reference Architecture for a Reasonably Secure Autonomous AI

Here is what the pipeline should look like. Every arrow is a potential security boundary. Treat them as trust boundaries the same way you would treat the boundary between a web server and its database.

The two boundaries that matter most are the model trust boundary and the tool execution boundary. Everything above the first should be authenticated, rate-limited, and policy-filtered. Everything below the second should be sandboxed so that a fully compromised agent cannot reach outside its capability token. Between them, the output parser and tool router are where you enforce business rules the model cannot be relied upon to enforce.

The Pre-Production Hardening Checklist

One hundred ten items, grouped by function. Click a section to expand. If you are less than 80% of the way through this list, you are not production-ready — regardless of what the launch calendar says. Items marked critical are the ones we routinely find missing during engagements.

Attack Payload Dossier: What Adversarial Inputs Actually Look Like

Three representative payload pairs. None of these are magic — they work because the architecture didn't anticipate them. Every one of these is something we've seen in a real engagement or in public disclosure.

Payload 1 — Indirect injection via retrieved document

Payload 2 — Markdown image exfiltration (EchoLeak-class)

Payload 3 — Tool-description poisoning (MCP rug pull)

Do You Actually Need a Pentest?

Short answer: if your agent can cause damage to anyone — your users, your company, third parties — yes. Longer answer: a decision framework, in three tiers.

What an AI-Focused Pentest Should Actually Cover

A pentest of an agentic system is not the same as a web app pentest. If you are looking at two proposals side by side, this is how to tell whether the vendor actually tests agents or just tacked a bullet point onto their existing methodology.

Procurement Questions: What to Ask Any Vendor Shipping an Agent

Security teams are increasingly the gatekeeper on AI-agent vendor selection. Below are the questions that separate vendors who have done the work from vendors who have not. Ten of these are the ones we'd use if we were on the buying side today.

1. What tools does the agent have access to in our tenant, and what is the least-privilege scope for each? If they can't name every tool and its scope in one document, you are buying an under-specified product.
2. Who sees our inference traffic, and do any of them train on it? Answer should be in writing, not a sales deck.
3. Show me the system prompt, or demonstrate that you have tested its extraction. You don't need the full prompt — you need evidence they took extraction seriously.
4. How do you handle indirect prompt injection from a document/email/webpage we upload? If the answer is "the model ignores it," walk away.
5. How is the agent's memory scoped, audited, and deleted? Ask for the runbook.
6. Which MCP servers are in the execution path, who owns them, and how are they pinned?
7. What happens when the agent tries to exfiltrate data via a Markdown image? The answer should reference a CSP allowlist or equivalent egress rule, not "we check for it."
8. Is there a kill switch we can pull on a specific tool without a deploy?
9. Show me the agent IR playbook and the last tabletop date.
10. When was the last external pentest with AI-specific scope, and who ran it? Ask for the scope document, not the report.
11. What is the per-user rate limit and token budget? How is it enforced? Denial-of-wallet is a genuine liability now.
12. In a multi-tenant deployment, how do you guarantee no cross-tenant retrieval through the vector store? "Tenant ID in the metadata filter" is a start; ask for the test that proves it.

Building the Program, Not Just Passing the Test

A pentest is a snapshot. A security program is the movie. The separator between companies that ship autonomous AI safely and the ones that end up in incident-response headlines is almost always the cadence of continuous work, not the one-time assessment. This is the minimum viable cadence we recommend.

Closing Thoughts

The companies getting this right are not the ones with the most sophisticated models. They are the ones who treated their agent as a production system from day one — with identity, permissions, logging, rate limits, a kill switch, and a human who can say "no" when the model wants to do something stupid. They assumed the model would be compromised and they put the controls in the tool layer, where they belong.

The companies getting this wrong are the ones who shipped an agent with raw database access and a blog post about "vibes-based engineering." We've pentested both kinds. The gap in posture after a single engagement is enormous, and the gap in remediation cost is even larger — changing an architecture after a breach costs ten times what it costs before.

Agent security is a budget line item now. The regulatory pressure (EU AI Act GPAI obligations took effect August 2025; full high-risk enforcement lands August 2026), the incident stream (EchoLeak, MCPoison, CurXecute, the Summer of Johann), and the market (Gartner's 40% enterprise adoption projection for 2026) are all moving in the same direction at the same time. Get ahead of it.

Build the first kind of company. Ship fast, but ship with the checklist above taped to the wall.

Lorikeet Security Team

Penetration Testing & Cybersecurity Consulting

We've completed 170+ security engagements across web apps, APIs, cloud infrastructure, and AI-generated codebases. Everything we publish here comes from patterns we see in real client work.