Measuring an AI Pentester: Precision, Rejection Reasons, and Cost Per Approved Finding

Every AI security product makes the same implicit promise: it gets better over time. Almost none of them can prove it, and the reason is mundane. The single most valuable signal these systems generate — a qualified human looking at a proposed finding and deciding whether it is real — is captured as a status flag, used to filter a list, and then thrown away.

Ours did exactly that. Approve and reject existed. They controlled what appeared in a client dashboard and drove a pending-count badge. Nothing aggregated them. Worse, our findings carried no record of which technique produced them, so even if we had aggregated the outcomes, we could not have attributed them to anything actionable.

This post covers what we built to fix that, the numbers it produced on first run, and the questions we think buyers should be asking every vendor in this category — including us.

Why Approve/Reject Is The Only Honest Metric

Security tooling has a long history of measuring the wrong thing. Finding counts reward noise. Coverage percentages measure what you ran, not what you found. Benchmark scores measure performance against a fixed corpus the vendor usually chose, and any system can be tuned toward a benchmark it knows about in advance.

Precision — the share of proposed findings a human confirms — has none of those problems. It is measured on real output, against real targets, judged by the person whose name goes on the report. It cannot be gamed by producing more findings, because every additional weak finding a reviewer rejects pushes the number down. It is, bluntly, the number a vendor least wants to publish and the one you most want to see.

Three questions follow from it, and all three were unanswerable for us six months ago:

• Precision per technique. Of our library of testing skills, which produce findings that survive human review and which produce pure noise? A skill at 0.9 and a skill at 0.05 look identical in a capability list.
• False-positive rate by model and asset type. When we change the model behind the agent, does output quality move? Without a per-model breakdown, a model upgrade is a vibe.
• Cost per approved finding. Inference and compute spend divided by approvals, by depth tier. Rejected findings cost real money to produce; this is the metric that tells you whether a depth tier earns its price.

The Prerequisite Nobody Budgets For: Attribution

Here is the part that surprised us. The measurement work is not the hard part. The hard part is that you cannot measure anything until each finding records where it came from, and that is a schema decision made long before anyone asks the question.

Our AI findings recorded title, description, severity, evidence, remediation, affected asset. What they did not record was the attack vector, the specific skill, the model version, or the depth tier that produced them. The vector label existed only in a separate engagement activity log, joinable to a finding by nothing better than fuzzy timestamp correlation.

So per-skill precision was not merely unreported. It was not computable — not with a clever query, not with an afternoon of data wrangling. The information had never been written down.

The fix was one migration adding indexed columns to the findings insert:

A day of work. It is the hard prerequisite for everything else in this post, and it is exactly the kind of change that never gets prioritised because it ships with no visible feature attached to it.

Making Rejection Mean Something

A reject button with a free-text field feels thorough and is analytically worthless. You cannot count prose. Six months of thoughtful rejection notes produce exactly zero decisions.

Rejection now requires selecting one of six fixed reasons, and the taxonomy is deliberately built so that each value points at a different owner and a different fix:

That last distinction is the one most teams get wrong. Four of these six reasons indicate a defect in the agent. Two do not. A metric that treats them identically will send you optimising the wrong thing.

The First Real Baseline

With attribution and the taxonomy in place, the analytics ran against production data for the first time. Here is the honest starting picture, sample size included:

14 approved, 5 rejected, remainder still in review. A baseline, not a benchmark.

We want to be careful about what this number is and is not. It is a genuine measurement of real output judged by real reviewers. It is also computed over a small sample, from a period before the engine rebuild, and it comes with an asterisk we cannot remove: every one of those findings groups as unattributed, because they predate the columns that would tell us which skill produced them.

So the first real number we have is simultaneously the last number of the old era. Its value is not the 0.74. Its value is that a month from now there will be a second number, computed the same way, broken down by skill, model, and depth — and the comparison will mean something.

We would rather publish a small honest baseline than a polished figure with no sample size attached. When a vendor shows you a precision or false-positive number, the follow-up question is always: over how many findings, judged by whom, on what date?

What The Signal Actually Drives

Metrics that do not change behaviour are decoration. Three mechanisms consume this data:

One piece is deliberately not built yet: a full evaluation harness with golden transcripts and schema validation gating skill edits. Today, a bad edit to a shared formatting instruction can degrade output across multiple surfaces, and the first signal is a human noticing. That is a real gap, it is on the roadmap, and we would rather name it than let the section above imply the loop is complete.

Where The Human Actually Sits

"Human in the loop" is claimed by nearly every vendor in this category and specified by almost none. Since the entire measurement loop above depends on human judgement, it is worth stating precisely what ours does.

• Every AI finding is written to a pending review state. It is excluded from reports, exports, client dashboards, and the findings list until triaged. This is enforced at the query layer on every artifact, not by convention.
• A Lorikeet Security pentester triages it. Approval means a person read the evidence and agrees. Not a second model, not a confidence threshold.
• Rejection requires a structured reason. As above. The reviewer cannot skip it.
• The disposition is data. It is stored, aggregated, and fed back into which techniques run by default.

The review gate had been real and enforced for a long time before any of this — and documented nowhere a client could see. We have since published the human-in-the-loop policy, a model card describing what the agent is and how it is evaluated, an emergency abort protocol, a data retention policy, and a subprocessor list. These are days of writing, not engineering quarters, and they answer procurement questions that no amount of engine work addresses.

Questions To Ask Any Vendor In This Category

1. What is your precision, and over how many findings? A number without a sample size is marketing. A vendor with no number has not built the loop.
2. Are rejection reasons structured or free text? Free text means the signal is being collected and discarded.
3. Does a finding record which technique produced it? If not, no per-technique quality analysis exists, no matter what the dashboard shows.
4. What happens to a technique that consistently produces rejections? If the answer is "nothing automatic," the product is not learning from your reviewers' work.
5. How do you validate a model change? Against a historical corpus, or against impressions?
6. Is accepted risk counted as a false positive? A surprisingly good tell for whether anyone has thought carefully about the metric.
7. What does an approved finding cost you to produce? Vendors who know this number have modelled their own economics. Vendors who do not may be pricing on hope.

We publish these knowing they will be asked back at us, which is the point. Related reading: Introducing: Lory AI Autonomous Tester describes the engine these metrics measure, and The AI Pentester That Couldn't Send a POST Request covers the external review that started all of it.

The Uncomfortable Conclusion

An AI security product that cannot measure its own precision is not a product that happens to lack a dashboard. It is a product with no mechanism for improvement, sold on the assumption that improvement happens on its own. Model providers do ship better models, and everyone downstream benefits — but that is the vendor's supplier improving, not the vendor.

The part that is genuinely yours is the loop: attribution, structured outcomes, aggregation, and a mechanism that prunes what does not work. It is unglamorous, it ships with nothing to demo, and it is the only durable advantage in this category. We started ours late. Publishing the baseline is how we make it expensive to quietly stop.

Lorikeet Security Team

Penetration Testing & Cybersecurity Consulting

Lorikeet Security helps modern engineering teams ship safer software. Our work spans web applications, APIs, cloud infrastructure, and AI-generated codebases — and everything we publish here comes from patterns we see in real client engagements.

Practical security, written by practitioners.