The AI adoption dashboard on my client's monitor said 94% compliance. She smiled. I did not.
She had set the target at 25 queries per user per week. The number on the dashboard was the number she had asked for. That was the problem.
Every executive I coach now runs a version of this dashboard. Almost none of them can tell me what fraction of the number is real.
The chief of staff who read the number and stopped looking
A chief of staff at a Stuttgart industrial group sat across from me in June. Six months into a €40M internal-assistant rollout. Her board deck opened with the compliance rate. The line went up and to the right.
I asked her to open the raw query log and sort by user. The top user in her sales org had 27 queries the previous week. Twenty-three of them were variations of "summarise this email." Two were the same prompt twice. One was a birthday poem.
She stared at it for a while. Then she said the quiet part: "So how much of the dashboard is like this?"
I did not know. She did not either. Nobody had asked. The number in her Monday board deck did not have a column for that question.
What the paper measured
Jie Gong and four colleagues at HKU's Faculty of Business and Economics ran the study I have been waiting for. A large Chinese medical-device firm, roughly 9,000 workers across 46 branches, mandated 200 queries per employee per month to its internal AI assistant. Under that target, 31% of queries were repeated or off-task and the counts bunched right at the threshold.
Then the firm halved the target to 100 queries per month, rolled out branch by branch. The authors ran a staggered difference-in-differences on around 44,000 employee-month observations.
Halving the target cut total queries by 28.69 per employee per month. Repeated or off-task queries fell by 25.97, about 90% of the entire drop. Non-repeated, work-related queries fell by 2.72, statistically indistinguishable from zero. In the sales subsample, monthly sales rose by 7%.
The estimated change in non-repeated, work-related queries was small and statistically insignificant.
The mandate was mostly measuring compliance theatre. Halving it took the theatre out and did not touch the work.
The chief of staff had spent two quarters defending a number that was three parts real and one part birthday poems. She could not tell me the split because the dashboard did not have a column for it. Neither does yours. Publish the composition before you defend the floor.