The short version. AI security tools have made it trivial to produce vulnerability findings. That is exactly the problem. The number of issues a tool reports tells you almost nothing about how many are real, and a long report full of unconfirmed claims does not reduce your risk. It transfers work to the one resource you cannot buy more of: a senior engineer's attention. The question that matters is no longer "how many did it find," but "how many did it prove."
The problem nobody wants to say out loud
Run a modern AI-driven scanner against a real environment and you will get a queue. Hundreds of findings, each written in confident prose, each plausible. Somewhere in that queue are the three that would have ended your quarter. The rest are noise: a version string matched against a CVE that does not apply, a "critical" that needs an authenticated admin to trigger, a co-hosted service that was never yours to begin with.
The tool has done the easy thing. A language model wrapped around a scanner is very good at generating findings and very bad at being accountable for them. Generating is cheap. Being right is expensive. When the two get separated, the cost does not disappear. It lands on a human who now has to reproduce, triage, and disprove each claim by hand.
Security teams have a name for what happens next. It is the same thing that happened to every noisy IDS and every over-tuned SIEM before it. Analysts learn that most alerts are wrong, so they start dismissing them. A 2022 USENIX study of SOC analysts put a number on the felt reality: teams describe drowning in alarms that are almost all false. Once a tool has cried wolf enough times, its true findings get dismissed with the false ones. The most dangerous outcome of an unverified tool is not the false positive. It is the real finding that gets waved away because the last ten were garbage.
Why "more findings" became a selling point, and why it is a trap
Vendors count findings because findings are countable. "24 times more issues than a single-pass scan" is a great slide. It is also, on its own, meaningless. More findings from an unverified process means more triage, more analyst-hours, more chances to dismiss the one that mattered. The metric rewards exactly the wrong behavior.
We know this because we measured it on our own system. In a controlled ablation of our pentest engine, turning the reasoning loop up produced a large jump in raw finding count. When we added a matched control that ran the same tools without the reasoning, most of that jump vanished. A lot of what looked like intelligence was just tool budget: more scans, more output, more to sort. The part that actually held up under scrutiny was much smaller, and much more valuable. If we had shipped the big number, we would have sold noise.
The shift: prove it, or do not report it
The fix is not a better model. It is a different contract. A finding should not reach a human until an independent, deterministic check has confirmed it. Not the model's opinion that it worked. A separate oracle, one the model cannot talk its way past, that fires only when the vulnerability is actually present.
This is a real engineering constraint, and it costs you recall. Some true findings will not clear the bar and will be held back or downgraded. That is the trade, and it is the right one. A report where every line is confirmed is worth more than a report where every line is a maybe, because the confirmed report is one a human can act on without re-doing the work.
Concretely, that discipline looks like this:
- Web and network findings are checked against the live target with an out-of-band signal the attacker controls but benign traffic cannot produce. If the planted witness does not fire, the finding does not ship. "The model said it worked" is not evidence.
- Memory-safety findings climb a ladder: a crash, then a sanitizer-typed fault, then proof the input controls the fault, then a working demonstration. The report states the highest rung it actually reached, and downgrades honestly rather than dropping to a lower-confidence claim dressed as a high one.
- Every finding carries its proof so the receiver can re-check it in seconds instead of trusting the sender.
How Nemesis Red does it
Nemesis Red is an autonomous offensive engine that drives the real toolchain, roughly 290 Kali tools over SSH, and reasons over the combined output. The design premise is inverted from most AI pentest tools: the report is only as good as the proof behind it, so proof is the gate. A finding that cannot clear a deterministic check does not appear in the report. A clean run reports nothing, which is a feature, not a failure.
The results we publish are the verified ones. On a labeled benchmark, validated precision held at 82.1 percent across 252 findings, using a deterministic annotator with no LLM acting as judge, so the number reproduces across re-runs instead of drifting with a model's mood. In a controlled factorial ablation, the thing that causally drove validated-finding quality was cross-tool evidence fusion, sharing facts between tools so the scanner gets pointed at the right service, not the reasoning loop replanning for its own sake (p < 0.001). And on live targets it has surfaced and reproduced real critical issues, including CVSS >8.0 CVEs, each with a working proof rather than a version-string guess.
We are deliberate about what we do not claim. We do not headline a fixed count of "zero-days," and we distinguish a reproduced known-CVE from a novel discovery. When a high-severity CVE turns out to be a version-based association on legacy infrastructure rather than an exploited path, we say so in the report. That honesty is the product.
What to demand from any AI security tool
You do not need to take a vendor's word, including ours. Ask for the following and watch how quickly the conversation narrows:
- Show me the proof, not the finding. For a claimed vulnerability, what independent artifact confirms it, and can I re-check that artifact myself without trusting you?
- What is your precision on a labeled set, and who labeled it? A recall or "coverage" number with no precision is a way to avoid the question.
- What does a clean run produce? If the answer is "it always finds something," you are buying a tool optimized to justify itself.
- Separate reproduced-known from novel. A tool that blurs the two is managing your perception, not your risk.
- What do you refuse to report? A vendor that cannot answer this has no bar at all.
The teams that win the next few years will not be the ones with the most findings. They will be the ones who stopped paying humans to disprove a machine's guesses, and started trusting a machine that only speaks when it can prove it.
Nemesis Red is the autonomous, proof-gated offensive engine from Nemesis Labs. It runs the real Kali arsenal, verifies before it reports, and ships nothing it cannot prove. nemesislabs.xyz