Main Content Area
Scale-related Pet-Peeves
Blog #28
Questionable Scale Unidimensionality
One of the criteria I use when reviewing scales is unidimensionality. In other words, do all of the items in a scale load on the same dimension in a factor analysis thereby providing basic evidence they are measuring the same construct? Unfortunately, even in our top scholarly journals, only a minority say anything about the unidimensionality of their scales and even fewer provide evidence of it. What should I do when no evidence is provided or when my judgement clashes with the evidence provided by authors?
In some cases, I see several items that I believe represent different constructs yet the authors claim that a factor analysis shows the items have loaded on the same factor. If in doubt, I normally defer to the empirical results but there are times when I am not convinced the items are measuring the same thing. One of the most typical scenarios is that researchers will combine a couple of items measuring attitude toward an object (It is a good brand) with a couple items measuring behavioral intention (I plan to buy the product). Although the two are related, there is no doubt in my mind that they are different constructs. In such a case, if the authors make no claim of the scale's unidimensionality, I am very likely to exclude the scale from the book because I cannot recommend it. But, when the authors say the items loaded on the same factor, I am conflicted about what to do.
Let me give a specific example. I was examining a scale recently and could not figure out what it was measuring. The items seemed to refer to different things or, at best, they referred to different points on some sort of continuum as in a Guttman scale. Not only that, but at least two of the items were almost completely opposite in meaning. I tried to rationalize the meaning of the items with the indication of reverse coding as well as the factor loadings but never could make sufficient sense of it all. Ultimately, I could not in good conscience recommend the scale for use and will not include the scale in the next volume.
An even more egregious situation occurs when researchers state the source of the scale items and, upon checking it out, I find the items were used in the source to measure different constructs. In such cases, it is not just my opinion anymore because the cited developer of the scale(s) has provided evidence that different sets of items were used to measure different constructs. If someone later mashes some items together that are known to have measured different things in past research, they have some explaining to do. Even if they claim their mashup is unidimensional, I am skeptical. In some cases, I may go ahead and include such a scale in the book but express concerns to readers in my review.
Finally, the use of factor analysis on a few scale items by themselves is cheating or, at least, isn’t much of a test. Instead, to really mean something, the items should be factor analyzed with other items, some expected to measure different but related constructs and some expected to measure constructs with which there should be little or no relationship. If the items in the focal scale still load together on the same dimension that would be much more convincing evidence of unidimensionality than running factor analysis just on the items in question.
The bottom line is that researchers should have a greater sensitivity to the items in their scales. Sure, I would like for them to test all forms of validity but they should at least have higher standards when it comes to judging face validity as well as testing unidimensionality.