Case study one: six engagements, twelve applications, 38 findings.
Lory is the AI pentester inside Talon. She runs scoped engagements rather than scans: you define what is in and out of scope per asset, she works the target, and a human reviews every finding before it reaches you.
Most AI security tooling is sold with adjectives. We would rather show you the finding list.
Over six engagements we pointed Lory at twelve deliberately vulnerable applications, and then at our own production platform. This post contains what she found, how she reached four of the more interesting conclusions, and, because it matters more than the headline number, an honest account of what these results do not prove.
The single longest engagement produced 26 findings in 39 minutes and 40 seconds, unattended, across seven applications. That is roughly one confirmed finding every ninety seconds.
The Setup
The targets were a set of deliberately vulnerable training applications, the kind used for hands-on security education, with names like mr-robot, splinter, abbys-lab and shuttle-booking. The bugs in them are planted. That matters, and we will come back to it.
Coverage was not limited to web applications. The same engine also produced findings against an Android manifest and an internal network host, which is the point of running one agent with a shared toolbelt rather than a different scanner per asset class.
On scope. Every engagement runs inside a boundary you set per asset: explicitly in-scope and explicitly out-of-scope entries. Lory works the target inside that boundary. This is not a nice-to-have for an autonomous agent; it is the thing that makes an autonomous agent safe to point at production.
The 39 Minutes
This is the actual discovery timeline from the longest run, straight out of the engagement record. Times are the moment each finding was confirmed and written.
Two things are worth noticing in that list beyond the pace. The first is that the findings get harder as the run goes on. The early entries are direct, unauthenticated code execution, while the last four are the ones that required reading code and reasoning about it. The second is the gap between 03:10 and 03:14. That is not idle time; it is the agent working a problem that did not resolve in one step.
Run this against your own assets
Lory tests your in-scope assets autonomously and files findings straight into your dashboard.
Four Findings Worth Reading
The count is the least interesting part of this. What separates an engagement from a scan is how a finding is reached, and four of these are worth walking through properly:
- A sanitiser that does not sanitise. An app filtered login input with
FILTER_SANITIZE_FULL_SPECIAL_CHARSbefore building a query. A scanner reads a recognised filter as a taint barrier and suppresses the finding. Lory worked out that this particular filter does not prevent SQL injection, and proved it, then found the same pattern in two more files. - Two dull bugs that add up to critical. An open redirect on a password-change endpoint, followed through to an unauthenticated shell. Remote code execution assembled from two issues that are unremarkable alone and would be triaged separately.
- Template injection through the database. Stored post content reaching Twig's
createTemplate(), written in one request and evaluated in another, with no single request that demonstrates the bug. - A private key that mattered. Not just a key-shaped string, but a second file naming its owner, a Dockerfile shipping it into the image, and a passwordless sudo grant for that same account.
Each of those is a case where a signature-based tool has a specific, explainable reason to produce the wrong answer. We pulled them apart with the real code and remediation in part two of this series.
What The Numbers Say
Across the lab set, findings clustered where you would expect from applications written to be broken, but the distribution is still informative.
| Application | Findings | Critical | High |
|---|---|---|---|
| marketer | 6 | 2 | 1 |
| abbys-lab | 5 | 2 | 2 |
| blogger | 4 | 1 | 1 |
| backdrop | 3 | 1 | 1 |
| shuttle-booking | 3 | 1 | 1 |
| mr-robot | 3 | 1 | 2 |
| splinter | 2 | 2 | 0 |
The most common weakness classes, by CWE, were information disclosure (CWE-200), SQL injection (CWE-89), protection mechanism failure (CWE-693), improper access control (CWE-284) and OS command injection (CWE-78). The full class list across the set:
- SQL injection: in login, in password change, and in authentication middleware
- OS command injection and remote code execution: four distinct paths
- Server-side template injection through stored data
- XML external entity injection
- Insecure direct object reference permitting arbitrary password change
- Broken authentication and session fixation
- Hardcoded credentials in source, Dockerfiles and compose files
- Plaintext password storage and unsalted SHA-512 hashing
- Weak file upload validation, directory listing, debug mode in production
Then We Pointed It At Ourselves
Finding planted bugs in training applications proves method. It does not prove the agent is useful against software written by people trying to get it right. So before asking anyone to trust Lory, we ran her against our own production platform, where nothing was planted and where we did not get to choose the outcome.
She found real issues, including a server-side request forgery flaw and a cross-session access-control bug. Both are remediated. That engagement, and what we deliberately are not publishing about it, is part three.
What This Does Not Prove
We would rather make the small honest claim than the large vague one, so:
Deliberately vulnerable applications are deliberately vulnerable. The bugs were meant to be found. A high finding count against training targets is a demonstration of method, not a measure of difficulty, and anyone quoting "26 findings in 39 minutes" as a capability benchmark, including us, should say what the targets were in the same breath.
Speed is not the interesting variable. Ninety seconds per finding sounds impressive and mostly reflects that these targets are dense with bugs. On real software the rate collapses, and it should. What survives from this run is the shape of the reasoning: the sanitiser analysis, the chain, the stored-input trace.
A human still reviews every finding. Nothing in this post reached a report unreviewed. That is a deliberate product decision and not a temporary state of affairs. An agent that files directly to a client is an agent that files its false positives directly to a client.
This is one agent on one set of targets. It is a case study, not a benchmark. We have published what an external review found wrong with this same engine, and we would encourage the same scepticism here.
How To Try It
Lory runs inside Talon, our client platform. You add an asset, set your scope rules (explicitly in, explicitly out) and start an engagement. Findings land in your dashboard alongside your human pentest and incident-response findings, with severity, age against your remediation SLA, and one-click push into Jira, GitHub, GitLab or Azure Boards.
If you would rather drive it yourself, Lory exposes an MCP server, so you can run engagements from your own agents with a bearer token.
See what Lory finds in your stack
Set the scope. She works the target. A human reviews every finding before it reaches you.
Start an AI pentest Explore TalonThe Lory case study series
- Part one: 26 findings in 39 minutes (you are here)
- Part two: Four findings a scanner would have walked past
- Part three: We pointed it at our own platform
Notes on method
- All figures are taken directly from the engagement records in our platform: 6 engagements, 12 applications, 38 findings, of which 10 critical, 11 high, 10 medium, 2 low and 5 informational.
- The 39-minute run is engagement timestamps from 02:47:28 to 03:27:08, producing 26 findings across 7 applications.
- Code excerpts are simplified for readability. The behaviour and the line references are as recorded in the findings.
- Findings against our own production platform are described in general terms only, and have been remediated.
