Most AI rollouts get measured on the metric the vendor wanted on the slide.
The one that would have embarrassed the rollout never makes it onto the deck.
A dashboard that looked magnificent
A CISO at a London asset manager walked me through her year-end board review last month.
She had rolled out a frontier AI vulnerability-finding capability back in March. Six months in, her internal disclosure count had more than doubled. High-impact findings were up around half again. Her slide looked magnificent, and the chief executive's note in the margin said "exactly what we hoped for."
I asked her the obvious question. Are you patching faster?
She paused. She said no. Patch velocity was flat. Then she added that external bug-bounty submissions on her stack were down. She thought that was good news. The hackers, she said, had stopped finding things.
I asked her where she thought those hackers had gone.
She blinked. Then she said, very quietly, that she had not looked.
What the paper actually measured
Thomas Mosk at Queen Mary University of London has a new paper doing the natural experiment I wish every vendor pitch had to survive. He studies Anthropic's Project Glasswing, which gave 13 US tech firms early access to a frontier vulnerability-finding model while policy kept Chinese and Russian firms out. The data covers 177,104 published vulnerabilities and 3,360 firm-months, difference-in-differences with the policy-excluded firms as the comparison group.
Three findings.
Internal disclosures at treated firms more than doubled. Glasswing × Post coefficient 0.784, Ibragimov-Müller p = 0.002. About 120 percent growth. High-impact issues up around 150 percent.
External bug-bounty researchers reallocated. The ones who had worked only on Glasswing code before launch reported 57 percent fewer vulnerabilities in that code afterward. They found about 2.5x more in other vendors' code.
Fix speed did not change.
The share of vulnerabilities exploited within a month of disclosure fell by two-thirds, from 1.4 percent to 0.5 percent. Defenders caught more things first. Patch velocity never moved.
The tool reshaped where effort went. Inside the firm, disclosures piled up. Outside the firm, the bug-bounty community picked up and moved next door. The downstream constraint stayed exactly where it was.
Every executive rolling out an AI capability should expect this pattern. The dashboard metric that would reveal it, time from first finding to resolution in your domain, is almost never on the rollout scorecard. The vendor would rather you track volume. The outside metric, where the external attention around your firm has reallocated, is not even on the list.
The CISO's slide showed what the tool surfaced. It did not show where the hackers were looking now. One of those numbers was on her deck. The other was moving without her.