Most AI quality dashboards track inter-annotator agreement as a success metric. Agreement up means the pipeline is working. On subjective tasks, that reading is doing damage I don't think most firms have priced.
The slide she was proud of
A head of trust and safety at a European consumer marketplace, on a 1:1 call last month, pulled up a quarterly deck. One slide was a Krippendorff's α chart. Last quarter 0.42, this quarter 0.68. The team had rewritten the annotator rubric and run a training sweep. Agreement rose. The board liked the slide.
I asked her what the annotators were labelling. Hate speech reports from users. Mostly immigration, religion, gender. Her annotator pool was Prolific workers screened for English fluency.
I asked her what she thought was happening when agreement went from 0.42 to 0.68. She said: cleaner rubric, better calibration, less noise.
I said: or less signal.
She looked at me like I was being clever. I was asking the question a paper that landed four days ago asks.
What the research says
Sheza Munir, Ramaravind Kommiya Mothilal and Syed Ishtiaque Ahmed at the University of Toronto ran a 2x2 experiment. Their sample was 104 Prolific annotators labelling 30 synthetic hate-speech items, for 1,040 annotations in all. 60 of the 104 annotators got a short pre-annotation stage first: a scenario vignette, an axis-guided conversation, and a first-person micro-narrative. The other 44 went straight to labelling.
Two engagement scores rose in the elicitation arm. Identity resonance went from 2.94 to 3.35 (p=.041). Harm sensitivity went from 2.78 to 3.12 (p=.029). The annotators were more in position to see what the rubric was asking about.
In the same arm, mean label entropy rose from 1.88 to 2.05 bits (p=.011). Pooled Krippendorff's α fell from 0.111 to 0.066. Agreement dropped by 40% in the arm where engagement rose. The gains concentrated in annotators with no personal connection to the topic. Non-lived-experience identity resonance rose 0.51 points (p=1.1×10⁻⁵) against 0.31 for lived.
"Elicitation raises engagement while lowering agreement."
The authors name the mechanism plainly. Positionality behaves like a state the interface either activates or leaves dormant. Training sweeps that push agreement upward on subjective tasks may be suppressing the thing the pipeline is meant to find.
My head of trust and safety has three KPIs on her deck. Two measure cost and throughput. The third measures agreement. None measure whether the annotators were in position to see the item.
That gap is where the dashboard lies to the room.