A randomized field experiment on Alibaba's Taobao platform tested what happens when an agentic AI hands a failing chat to a human. The humans rescued the technical failures and quietly underperformed on the emotional ones, sending fewer messages and pulling back exactly when frustrated customers needed them most.
Why this should bother executives
This is not a paper about AI quality. It is a paper about human quality after AI shows up.
The Alibaba researchers ran one of the cleanest tests I have seen of the assumption every executive is making right now: that a human-in-the-loop will catch what the AI misses. The data says it depends on what kind of miss.
When the AI failed for a technical reason (something beyond its capability), the human worker leaned in and the outcome held. When the AI failed for an emotional reason (a customer expressing frustration), the human worker leaned out. Fewer messages. Smaller share of the chat. Less proactive information seeking. The exact behaviors that recover an unhappy customer.
Customer ratings dropped only for AI-eligible chats, not for the chats workers owned start to finish.
What I keep seeing in my coaching room
Last month I sat with the COO of a mid-market SaaS company that had rolled out an AI support agent. Six weeks in, his deflection numbers looked beautiful. His CSAT had slipped below where it sat before any automation.
He walked me through the dashboard. Chats his team picked up after the bot escalated were rated lower than chats his team had handled cold three months earlier. Same people. Same training. Worse outcomes.
I asked what his team felt when a bot escalation hit their queue. He paused. "Like cleanup," he said. Like the interesting part of the work had already happened and they were inheriting the mess.
That word cleanup is the whole problem. A frontline worker who owns a customer from minute zero sees a chance to recover the relationship. A worker handed a frustrated customer mid-meltdown sees a job no one wants. The role changes. The energy changes. The dashboards do not catch any of it until churn does.
I am Fatjon Tony Kalemaj, and most of my coaching room this year has been some version of this story.
Most leaders rolling out agentic AI are budgeting for the technology and assuming the humans will absorb the rest. The Alibaba paper is the first hard evidence that they will not, and the gap shows up in the customer moments that matter most.
The instinct after reading this is to retrain the humans. That is the wrong move. The handoff is the product now, and it has to be designed.