Main Content Area
Scale-related Pet-Peeves
Blog #33
Poor Descriptions of Scales*
In my work reviewing scales, the primary task is to describe the measures. Well, I should say try to describe them because I encounter numerous obstacles in the process. Many of the challenges have been discussed in previous pet-peeve blogs I have posted over the years. In this post, I will focus on information given in articles that is confusing and incomplete with regard to exactly WHAT a scale is measuring.
Here are the types of problems I deal with when trying to describe scales.
- Scale items are paraphrased: The critical component in describing a scale is the items themselves. In other words, what are the questions, statements, semantic-differentials, etc. that compose a measure? The problem is that far too often, items are not given verbatim but rather they are paraphrased, e.g., “the scale had three items that asked how good, happy, and pleased participants felt after watching the ad.” While that is a description, it is is not very helpful to researchers who want to borrow the scale for use in their own studies. As with me, readers must guess the actual phrasing used in the questionnaire or track down the authors, request the items, and hope the authors respond.
- Not all scale items are provided: Another type of incompleteness occurs when authors only provide one or two items in the article rather than the entire set. I have seen this happen with scales the authors have created as well as with well-known scales that the authors have decided to shorten. As with the above, others who read about the scale and want to use it must try to get the missing information from the authors.
- Scales are described as “adapted”: Yet, a third type of incompleteness occurs when authors say they “adapted” a well-known scale but don’t describe how it was changed. I bemoaned the misuse of that term in one of my earliest pet-peeve posts. Even when authors use “adapted” appropriately, readers should be told HOW the scale was modified. For example, were some items dropped from the full scale, were items rephrased for a different context, or was there some other type of change? Going beyond the HOW, knowing WHY authors believed it necessary to modify a scale is useful to know as well. There could be good reasons for “adapting” a scale, but my point is that it should be stated.
- Constructs are ill-defined or not defined at all: There are many variations of this problem and so much can be said about them, but I will keep it brief here. Authors sometimes refer to a scale rather generally and the construct is not clearly identified, if at all. A clear example that comes to mind is when authors describe a scale as an “evaluation index” and provide no other description. In most cases these are measures of attitudes about some object or action but the name of the construct is not used. Another reason it can be difficult to identify what a scale measures is when the items in a scale do not align with the construct’s name and/or its definition. An example of this is when authors say they are measuring “purchase intention” but the items do not state any intention but rather attitudes toward the object or willingness to try it. (See my past blog about that issue.) The problem is compounded further when items in a set have been used to measure various different constructs. After all these years it still appears some authors believe that if a set of items they throw together has a high alpha then it is tantamount to being unidimensional and "good enough" for scholarly research.
Despite this depressing state of affairs, the good news is that the greater use of web appendices has given authors the space to provide more information about their scales. The excuse of not having the space in one’s article for listing construct definitions, scale items, factor analyses, and related information is not acceptable anymore. While some authors have used that space well, others have not. I call on editors and reviewers to raise their standards and require authors to provide complete information in published articles and/or web appendices as a requirement for publication acceptance.
-----------------------
* Specific articles could have been cited here but are not because of the potential embarrassment it could bring to the authors. My purpose is not to embarrass them but to urge them along with the journal editors, reviewers, and other scholars to raise standards of describing measures.