Every AI-productivity number I have been shown this quarter is real. The scores go up. What no one in the room can tell me is whether their team still knows when it is wrong.
That second half is the one being sold as free.
A chief underwriter in Zurich, in June
A chief underwriter at a European specialty insurer walked me through her Q2 dashboard. Her junior team's decision quality on complex marine risks was up sharply since they had started running cases through an internal LLM copilot. The chart looked exactly like the pitch her vendor made in January.
I asked her something simpler. When her juniors mark a case as uncertain, is the mark still landing on the right cases?
She looked at me for a while. Her senior reviewer had raised the same worry the week before. Escalation volume had dropped. The escalations that did come up were mostly wrong ones.
Her productivity chart was measuring one thing. Her senior reviewer was watching another.
What Aalto just measured
Fernandes and colleagues at Aalto ran the head-to-head. 366 US adults, between-subjects, on a 40-item cognitive battery of matrix reasoning, mental rotation, letter strings and syllogisms. Same battery, same GPT-5.6-Luna, as the Welsch paper I wrote about a week ago. Two arms of one research programme.
The Human+AI group beat the Human-alone group by +7.22 points, g=1.20, p<.001. That is the productivity number the vendors are quoting.
The metacognitive number is what they left out. Per-item confidence used to sort right answers from wrong ones at an AUROC of 0.649 unaided. With the AI, it fell to 0.564. Group difference −0.085 [-0.110, −0.060], g=−0.71, p<.001. The confidence gap between correct and incorrect answers collapsed from 0.12 to 0.04. The Dunning-Kruger quartile contrast widened from 8.87 points to 13.58. The AI-assisted group placed itself at the 59.95th percentile when the true mean is 50.
AI made the team better at the task and worse at knowing when it was wrong.
Signed estimation error was identical across the two groups (g=0.00). Over- and under-estimates cancelled in the aggregate. The dashboard sees nothing. The miscalibration sits at the bottom quartile, which now overestimates its own score by 9.38 points, against a top quartile underestimating by 4.20.
The people in the room whose confidence is hardest to challenge are now the least calibrated. The people who would flag a bad answer for you are the ones sounding the least sure. Ask your vendor for the AUROC number on the same task before you scale the productivity one.