siu logo

office of scale research

Department of marketing

OSR

Main Content Area

Scale-related Pet-Peeves

Blog #21

 

Simplistic Metrics Abuse

What a wonderful time we researchers live in when it is relatively easy to get data, sometimes in real-time, that allows us to know so many details about what our customers are doing. My beef is not with this technology nor the skills to use it in managing a company’s marketing activities. Instead, my beef is that even though this information helps us understand what customers did/are doing, it does not tell us why they did it or what they will do in the future. For those things, we need to get inside customers' heads. My fear is that too many researchers either do not understand this or minimize its importance.

For example, if you manage a website, you probably know from your analytics that during a certain period, some number of customers visited certain pages, spent a certain amount of time on a page, and a certain percentage of them bought something. But, what you do not know is how many of them intend to shop there again in a certain period of time. You do not know why some shoppers left the site without buying anything. Further, do not know why some part of the market has heard of your site but has decided it is not worth visiting. There are thousands of other important questions you could ask but behavioral data can not provide the answers. Instead, you need to get inside your shoppers’ heads.

The next step is to realize that, while some of these questions might be answered acceptably in a quick-and-dirty survey, examination of more complex variables and relationships requires more precise instrumentation. Further, simply asking one question per psychological variable is insufficient. That type of measurement is imprecise and/or inaccurate. It is akin to a carpenter who needs to cut some trim for a room, its doors and its windows, only measuring one of the room’s dimensions. Worse yet, assume that the carpenter uses a nylon rope to measure that one dimension within the room. Given the nature of that rope to stretch and contract, it is inconsistent in determining length even for the same dimension. Even if it was consistent, it is just one of several dimensions that need to be assessed in order to adequately understand what is needed for the room.

My point is that some variables and relationships are so important to you and your business that you should want the most accurate way of measuring them that you can afford. Too few researchers seem to understand that the tool for the job is a multi-item scale. Please see previous blogs here for more information about multi-item measures. Suffice it to say, using a score based on several (3-5) of the “right” items can greatly improve accuracy.

The bottom line is that researchers who are trying to measure something should use the right tool for the job. Behavioral metrics have their place but they are not the right tool to use in addressing many research questions where the answer is psychological. That is like a carpenter trying to gauge the dimensions of a particular room in a house from a photograph of the house’s roof seen using Google Maps. Likewise, 1-item survey questions have their place but they are woefully inadequate in most circumstances that involve what consumers think and feel.

If a mentor, a teacher, or even a stat-jock has told you that one-item measures of attitudes and emotions are accurate, I am challenging that notion. Again, to use the carpentry analogy, it is like a home owner being told by his carpenter that he doesn’t need to measure all the walls of a kitchen before cutting trim for it because he can do a “good enough” job with just one measurement. Well, would you let a carpenter with that mind set renovate your house? If not, why would you let a survey builder use sloppy, kindergarten quality measures that provide imprecise information about your market?

There is a better way! The good news is that researchers do not have to guess about the quality of their measures nor use quick-and-dirty measures that provide misleading information. Yet, at the other extreme, they probably do not have to go through the effort and expense to develop high quality measures. Instead, they can build upon the work of scholars who have created thousands of multi-item measures over time and who have also provided information about their measures’ precision and accuracy. I urge marketing researchers to learn about this work I am referring to and build upon it. There are various ways of doing this and the materials offered by the Office of Scale Research as well as MarketingScales help make it easy.