Main Content Area
Scale-related Pet-Peeves
Blog #31
Bogus Measurement of Satisfaction
Have you ever been presented with a satisfaction questionnaire and then been instructed, probably by the person for whom the survey is at least partially about, that it is VERY IMPORTANT to give the highest possible score? This has happened to me several times over the years. Let me give an account of what happened to me a few months ago. Maybe it will jog your memory of something similar that has happened to you.
A service tech came to my house to work on the satellite system. He provided service for a large, well-known company that was purchased a few years ago by an even larger telecommunications giant. (The logo of the giant is on the rating form I was given.) When the tech was finished with the work, he asked me to fill out two rating forms, one before he left and one that would come later in a call from the telecom. For the first one, he stood right next to me and watched closely as I filled out several questions about his service. He made it clear that I should give him the highest score because any other number implied failure. At the end of the form was a space where I was to sign my name. The second form, which I still have, just had one overall satisfaction question and urged me to give a 10, stating that anything other than 10, such as a 9, implied failure. (By the way, I recently learned of a chain of luxury resorts that routinely uses a similar process.)
What’s wrong with this?
- Telling customers what score to give makes raters feel uncomfortable and possibly irritated or even angry.
- Providing such feedback should be anonymous or, at the very least, the personal identification should be optional. Further, when the rater is being closely watched by the ratee, there is further pressure on the rater to not give a completely candid response, particularly if the response is negative.
- What’s the point of a 10-point scale if the customer is told the only thing that matters is the highest score? In essence, the company seems to treat the measure as a crude 2-point scale (satisfied/dissatisfied) which is inconsistent with the range of people's beliefs and feelings.
- Because of the social pressure and lack of anonymity, this process produces more 10s than otherwise would have occurred. That highly skews the data, biases interpretation of the results, and masks underlying negative beliefs and emotions that customers really have. When management is provided with such a rosy picture, actions may not be taken to remedy customer concerns. By not addressing them, even more customer dissatisfaction is likely.
- When people are coerced into giving the same high score, that has a negative impact on the analyses conducted to find relationships which management should be interested in understanding. Statistically, when there is little variance in the data, it makes it difficult to find relationships though they may actually exist. For example, let’s say that with perfect data we could easily determine that the later a service person arrives compared to what was expected, the worse the overall satisfaction. But, when most customers give the same high score then you will find little correlation between arrival time and satisfaction. Yet, THAT IS NOT AN ACCURATE UNDERSTANDING OF THE RELATIONSHIP!
WHO is authorizing these processes? Is it just a lowly staff member here and there who does not understand measurement, statistics, and the ethical implications of what he/she is doing? Alternatively, are there people in research or customer service who believe this is an appropriate way to measure satisfaction? Another possibility is that there are managers who are pressuring well-trained researchers to do something that is inappropriate. Whoever you are, I AM CALLING YOU OUT!!! If there is anyone out there who can justify this practice, please try do so. Until you do that, my position is that when a company uses such a rating system it gives management a very distorted view of the business. Further, if you use such methods, not only does it irritate customers, but it leads to bad word-of-mouth and its subsequent negative impact on future business. Finally, if a company knowingly uses distorted satisfaction ratings in its promotion then the fraudulent practice could have legal ramifications.