Case study two: the engagement where we could not curate the results.
Every security vendor's demo works. That is what makes demos worthless as evidence.
In the first two posts of this series we published a Lory engagement in full: 38 findings across twelve applications, and four of them examined in detail. We were careful in both to say the same thing: those targets were deliberately vulnerable training applications. The bugs were planted. Finding planted bugs demonstrates method, and nothing more.
So the obvious question, and the one we would ask a vendor: what happens when you run it against software that people actually tried to get right?
We ran it against ours.
The engagement we could not curate
Before asking anyone to trust Lory with their infrastructure, we scoped her against our own production platform, the same one our customers log into.
This is an uncomfortable test to run, for a reason that is worth stating plainly: we do not get to choose the outcome. Against a training target, a disappointing run can be quietly discarded and re-run with different settings. Against your own product, the findings exist whether or not you like them, and the only decision left is whether you are honest about them.
She found real issues. Nothing was planted. They were ours.
An encoded address slipped past a private-address denylist, letting a request reach somewhere it should not have. The mechanism is the classic one: the check inspected the textual form of the address a user supplied, and an IPv4 address can be written several ways.
A write operation that should have been bound to one session could be performed against another. An object identifier was trusted without confirming it belonged to the caller.
Both have been remediated.
What we are not publishing, and why. No endpoint paths, no parameter names, no bypass technique. The findings are fixed, and "it is fixed" still is not a good enough reason to hand out a working method for a service anyone can reach. Detail like that survives the specific bug. It generalises to similar patterns elsewhere in the product, and to anyone probing for a regression. Describing the class is useful. Publishing the recipe is not.
The SSRF lesson is worth generalising
We will not describe our specific case, but the underlying mistake is common enough to be worth writing down, because a great many applications defend against SSRF the same wrong way.
The instinct is to validate the address as a string: reject anything that starts with 10., 192.168., 127., and so on. That check is looking at the wrong thing. An IPv4 address has several legal textual forms: dotted-decimal, hexadecimal, octal, and a plain 32-bit integer. A great many HTTP clients and resolvers will happily accept all of them.
And a string check cannot see through DNS at all: a hostname you control can simply resolve to a private address.
The defence is to stop checking strings. Resolve the hostname first, then test the resulting binary address against the reserved ranges, and re-check after every redirect, because a permitted host can redirect to a forbidden one. Where the requirement allows it, an allowlist of permitted destinations beats any denylist, because a denylist has to be right about every possible encoding and a allowlist only has to be right about the ones you want.
Run the same test on your platform
Scope it per asset, in and out. She stays inside the boundary you draw.
Why scope rules are the whole product
Pointing an autonomous agent at your own production platform concentrates the mind on one design decision: the boundary.
In Talon, scope is defined per asset, explicitly in and explicitly out. It is not a suggestion in a prompt or a line in a system message; it is a rule the engagement is bound by. That is not a feature we added for marketing symmetry with human pentests. It is the thing that makes an autonomous agent safe to aim at anything you care about.
The second control is just as boring and just as important: a human reviews every finding before it reaches a client. This is a deliberate product decision rather than a temporary limitation we intend to remove. An agent that files directly to a customer is an agent that files its false positives directly to a customer, and a false positive costs an engineering team the same hour a true one does.
What this does and does not prove
It does prove the findings were not planted. That is the entire reason we ran it. Nobody seeded our production platform with an SSRF bypass to make a blog post work.
It does not prove completeness. Lory found these. We have no way to state what she did not find, and neither does anyone else running this kind of exercise. A clean run is not evidence of a clean product. It is evidence of a clean run.
It does not replace human testing. We sell human-led penetration testing and we still buy it for ourselves. Lory runs continuously between engagements; she does not substitute for a tester with a research agenda and a grudge.
We have published our failures too. When an external review found that this same engine could not send a POST request, because of one enum in one tool schema that silently disabled half its skill library, we wrote that up in as much detail as this. A vendor who only publishes the flattering runs is not giving you data.
The whole claim, in one sentence: we built an AI pentester, aimed it at our own production platform before we aimed it at anyone else's, it found real problems in our product, and we fixed them.
If you want to run the same test
Lory lives inside Talon. Add an asset, set the scope rules, start an engagement. Findings arrive in your dashboard alongside your human pentest and incident-response findings, with severity, age against your remediation SLA, and one-click push into Jira, GitHub, GitLab or Azure Boards.
If you would rather drive it from your own tooling, Lory exposes an MCP server, so you can run engagements from your own agents with a bearer token and your own scope rules.
Test yourself before someone else does
You draw the boundary. She works the target. A human reviews every finding before it reaches you.
Start an AI pentest Explore TalonThe Lory case study series
- Part one: 26 findings in 39 minutes, the full results
- Part two: Four findings a scanner would have walked past
- Part three: We pointed it at our own platform (you are here)
Notes on method
- The findings described were produced by a Lory engagement scoped against our own production platform, and have been remediated.
- Endpoint paths, parameter names and exploitation detail are deliberately withheld. The vulnerability classes are described in general terms.
- The IPv4 encoding examples are generic illustrations of the class, not our case.
