The AI Pentester That Couldn't Send a POST Request

We commissioned an external review of our own AI penetration testing agent. The reviewer's central claim was blunt: the engine cannot do what its skill library assumes. We validated all thirty claims line by line against our own source code. Twenty-three were confirmed outright. Four were confirmed and understated — the reality was worse than the reviewer described. Three were partly right about a real gap they had mis-characterised. Four were simply wrong.

This post is about the single finding underneath most of the others, because it is the kind of failure that is very easy to ship, very hard to notice from the outside, and — we suspect — extremely common in agentic security products right now.

One Enum, Half a Product

Our agent's HTTP tool declared its method parameter like this:

That is it. That is the whole thing.

The model behind the agent could reason about testing a login form perfectly well. It had detailed written methodology for mass assignment, for cross-site request forgery, for insecure deserialization, for GraphQL mutations, for business logic abuse, for race conditions, for file upload handling. It could plan those tests. It could describe them. It simply had no way to perform them, because the only tool that spoke HTTP could not express a request with a body, and could not use a verb that changes state.

Everything downstream of that constraint was unreachable regardless of anything else we fixed:

Roughly half our web skill library and effectively all of our authentication skills were blocked by two lines of JSON schema. The fix was a schema change and a few curl flags — two days of work, including the safety rules that had to come with it. Nothing else on our entire roadmap had a comparable ratio of impact to effort.

The Audit Method: Claim by Claim, Against Source

The review arrived as a list of capability claims. We did not respond to it as a document to argue with; we treated every claim as a hypothesis and went to the source code to confirm or refute it, recording a verdict and the specific file that justified it.

We used four verdicts, and the two beyond simple true/false turned out to be the valuable ones:

The "wrong" verdicts were the most instructive, because all four followed an identical pattern. The reviewer looked at what AI engagements could produce and concluded the capability did not exist. In every case the capability existed, was mature, and was wired to a different part of the product. Reports, structured CVSS, phishing campaigns, and ticketing integrations were all real — and all unreachable from an AI engagement.

The diagnosis that reframed the roadmap

Once every claim had a verdict, one sentence covered nearly all of them: we had a mature platform and an engine wired to almost none of it.

That is a much better problem to have than the alternative, and it is a completely different problem from the one the review appeared to describe. The review read like "this product is thin." The source said "this product is thick in the wrong place." Most of the work that followed was integration, not invention — teaching the report builder a third engagement type, adding columns to a findings insert, letting an existing scope enforcer be called from a new place.

What Got Built

The rebuild ran in a single intensive push and shipped to production on July 27, 2026. The toolbelt went from nine tools to sixteen.

The engine also got its first tests. It had none. It now has multiple suites totalling over 150 checks that run without a database, an API key, or a container — which meant we could run the full suite against production immediately after deploying to it.

Three Bugs Worth Publishing

1. The CVSS extractor that read the spec version as the score

A CVSS v3.1 vector string begins CVSS:3.1/AV:N/AC:L/…. Our new extractor, searching that string for a numeric base score, found 3.1 and stored it. A 9.8 critical was recorded with a base score of 3.1.

The reason this survived testing is worth dwelling on: severity still looked right, because severity is set separately by the model. Every dashboard, badge, and sort order looked completely normal. Only the new queryable score field — the one nothing rendered yet — was wrong. It was caught by running against real production findings rather than fixtures, and it now has a nineteen-check regression suite.

2. A regular expression that a JIT compiler miscompiled

While fixing the extractor we hit a pattern of the form (?:A|B)[^x]{0,n} that failed to match anything at all under PCRE2 10.47 with JIT enabled, while behaving correctly on 10.44. Production runs 10.44 and was unaffected, but a regex that works on your server and silently fails on a newer one is a genuinely nasty class of bug. We removed the construct and scanned the entire repository for other occurrences.

3. A migration written in the wrong dialect

Our migration used ADD COLUMN IF NOT EXISTS — valid MariaDB, rejected outright by MySQL 8.0, which is what production runs. Development and production disagreeing on database engine is an old story, and the durable fix was not to memorise the difference but to stop relying on dialect-specific DDL entirely: each change is now guarded by an information_schema lookup, which is portable and safely re-runnable on both.

Hardening That Came Out of the Same Review

Two scope-enforcement improvements are worth describing because they generalise to any system that decides whether a URL is allowed.

Target normalisation. Our original parser stripped the scheme and path and stopped there. It could not correctly handle userinfo, non-default ports, bracketed IPv6 literals, trailing dots, or internationalized domain names. A URL of the shape https://[email protected]/ normalised to a string that matched no rule — which happened to be safe, but safe by accident is not a security property.

Fail-closed matching. The rule now is explicit: a target the enforcer cannot confidently parse matches no rule at all and is refused. Previously an unparseable target could still be prefix-matched against the raw string, which is precisely the wrong direction to fail. Both behaviours are covered by tests, because scope enforcement is the one component where a regression is not a bug but an incident.

The halt checkpoint. The engine checked for cancellation once per orchestration loop. One loop iteration can invoke two long-running scanners back to back, so a stop issued mid-pass could go unhonoured for ten minutes. Cancellation is now checked at every tool boundary and fails closed if engagement state cannot be read — which is what makes a sub-minute abort guarantee honest rather than aspirational.

How To Evaluate Any Agentic Security Tool

If we could hand a buyer one page, it would be this. These questions are answerable in a demo, and the answers are hard to fake.

1. Which HTTP verbs can the agent issue, and can it send a request body? If the answer is vague, ask to see the tool definition. This one question would have exposed our entire root cause in thirty seconds.
2. Can it hold an authenticated session across an engagement? Follow up: how many simultaneous identities? Without two, it cannot test a privilege boundary.
3. Can it discover surface it was not given? A crawler, a bundle parser, a spec ingester. If it only tests the URLs you supply, you are paying for coverage you have to produce yourself.
4. Do findings from one phase inform later phases? If each pass is independent, you get a list, not an engagement.
5. What happens when a scope check is ambiguous? "Fail closed" is the only acceptable answer, and it should be demonstrable.
6. What does the agent refuse to do? A product with no refusals has not thought about failure modes. Ours refuses four asset types outright rather than run a shallow test and report a misleading clean.
7. Who reviews findings, and is rejection structured? Free-text rejection cannot be aggregated, so it cannot improve anything.
8. Can it produce the artifact your auditor asks for? Report, attestation letter, and a machine-readable export. A finding trapped in a dashboard is not a deliverable.

Why Publish This

Writing publicly that our own product could not send a POST request is not an obvious marketing decision. We are doing it for two reasons.

The first is that the security industry asks its customers for exactly this behaviour. We tell companies to disclose, to write honest postmortems, to describe what actually failed rather than what sounded manageable. A security vendor that will not apply that standard to itself is asking for a courtesy it does not extend.

The second is more practical: this failure mode is a structural risk in every agentic product being built right now, ours included. The skill library grows faster than the toolbelt because writing methodology is cheap and building tools is expensive. The model is fluent enough to make the gap invisible in the output. If you are building in this space, the check is simple and worth running today — take your ten most advanced documented capabilities, and for each one, name the exact tool call that performs it. Ours failed that check on more than half.

Related reading: Introducing: Lory AI Autonomous Tester covers what the rebuilt engine does, and Measuring an AI Pentester covers how we now measure whether it is getting better.

Lorikeet Security Team

Penetration Testing & Cybersecurity Consulting

Lorikeet Security helps modern engineering teams ship safer software. Our work spans web applications, APIs, cloud infrastructure, and AI-generated codebases — and everything we publish here comes from patterns we see in real client engagements.

Practical security, written by practitioners.