Describe a time the numbers and the stories you were hearing did not match. What did you do?

I was an analyst at a regional insurer, supporting the claims call center. We had rolled out a target to bring average call time under six minutes, and the dashboard said it was working. Call time fell from eight minutes to five and a half in one quarter. But the team leads kept telling me customers were angrier than ever. I didn't want to pick a side, so I went looking for a number that would be true if the team leads were right. I dug into some other metrics and found that repeat calls had gone up a lot. Then I listened to calls to find out why. Some reps were rushing customers off the phone to hit the target. The reps weren't being lazy. They were doing what we'd asked for. I took it to my manager and the head of claims. We replaced the call time target with one for issues resolved on the first call.

I was an analyst at a regional insurer, supporting the claims call center. We had rolled out a target to bring average call time under six minutes, and the dashboard said it was working. Call time fell from eight minutes to five and a half in one quarter. But the team leads kept telling me customers were angrier than ever. I didn't want to pick a side, so I went looking for a number that would be true if the team leads were right. If calls were shorter but problems weren't getting solved, people would call back. So I pulled repeat calls within seven days, which nobody was tracking. They had gone from 18% to 31%. Then I listened to calls to find out why. I picked 40 recordings from the reps with the shortest times. In about a third of them, the rep transferred the customer or promised a callback right around the five-minute mark. The reps weren't being lazy. They were doing what we'd asked for. I took it to my manager and the head of claims. I brought one chart with call time and repeat calls on the same axis, and I played three of the recordings. We replaced the call time target with one for issues resolved on the first call. Within two months repeat calls were back to 20%, and average call time settled around seven minutes. Since then, when a metric improves fast, I ask what it would look like if people were gaming it, and I go check that first.