I am often asked some version of this question:
We just saw the [Temkin Experience Ratings/Temkin Group’s NPS benchmark/Forrester’s CXi/JD Powers/The ASCI] and it is completely different from what our internal data is telling us. How should we reconcile the two data points?
Given that I created several of those industry measurements, it’s fair for people to ask me that question. Here’s my answer…
Different Measurement Systems Deliver Different Results
There is no perfect or “ultimate” customer measurement system, since we can never know what every customer is thinking at every moment in time. So all measurement systems are, by definition, somewhat flawed. This is an important point because we need to let go of the desire to identify which one has the “right” information.
Every customer measurement system can differ along a number of dimensions. In particular, these are often key differences between your internal system and industry benchmark studies:
- Who’s being surveyed? Temkin Experience Ratings, for instance, asks questions to randomly selected consumers who have interacted with companies. Internal measurements are often less random, since they may neglect people who haven’t provided contact information or people who are no longer customers.
- When are they being surveyed? Temkin Experience Ratings, for instance, asks questions during January. Internal measurements may ask questions throughout the year, after specific interactions, or during specific periods of time.
- What’s being asked? Temkin Experience Ratings, for instance, asks three questions, covering Success, Effort and Emotion), on a seven point scale. Internal measurements can be almost anything, including Net Promoter Score that is standardized on an 11-point scale, but we’ve seen companies use 7- and 10-point scales as well.
- How is the metric calculated? Temkin Experience Ratings, for instance, is an average of net scores for Success, Effort, and Emotion, which are calculated by taking the percentage of 6s and 7s, and then subtracting the percentage of 1s, 2s, and 3s. Internal measurements may be average scores, they may be segmented by different customer groups, they may be top box or top 2 box, or just about anything.
Given that internal measurement systems are typically different than industry measurements across those four items, it shouldn’t be a surprise that they often deliver different results.
My Take: Rely on Your Internal Data
Instead of trying to find which one of the metrics is correct, I recommend that you:
- Understand the difference between the internal and external measurement systems (starting with the four items above).
- Learn what you can from each of them. Maybe the Temkin Experience Ratings shows that you are lower with the general population, but your data shows that you are really doing well with high-value customers.
- Improve your internal measurement system. Most companies we’ve seen have significant opportunities for improving their internal customer measurement systems. Make sure the focus is on building a system that drives improvement, not one that just keeps score.
- Whenever you can, rely on your internal data. Why? Because you can do more segmentation of the results, track changes to specific customers over time, and go deeper into questions about what’s driving the data. These are all things you may need to drive improvements.
The bottom line: Focus on improving, not on reconciling metrics.
This blog post was originally published by Temkin Group prior to its acquisition by Qualtrics in October 2018.