Main Content Area
Scale-related Pet-Peeves
Blog #35
Combining Items From Measures of Different Constructs
What is your opinion of the following scenario: a researcher decides to create a multi-item scale and does it by cherry-picking items from extant scales that measure different constructs? Does that sound appropriate to you? Well, it doesn't to me.
I have seen this done far too many times. Keep in mind that the instances I have seen and am concerned about were not done by untrained doctoral candidates or practitioners. The actions were taken by professors at well-known universities. Their articles were published in several of the most revered scholarly journals in the field of marketing: Journal of Advertising, Journal of Consumer Research, Journal of Marketing, Journal of Marketing Research, and the Journal of Retailing.
What is going on? Why are well known professors at well known schools getting papers published doing something that, on the surface at least, should not be done? What is the rationale for such behavior by authors and journal reviewers?
Let me give five specific examples of this behavior from the last ten years that illustrate what I think is unacceptable if we are suppose to be scholars who care about the quality of our measures. Since I am reluctant to say who the authors are, I will describe what they did with just enough detail that readers should understand the problem.
1. From an article in the Journal of Consumer Research, the authors drew some ideas from one researcher's measure of intention to do X and then merged them with items from another set of researchers' measure of "interest" in X. Are intention and interest the same construct? The authors apparently assumed so. They were so certain that they did not provide theoretical justification of their action nor did they provide evidence that the items loaded on the same construct.
2. Reference is made in a Journal of Advertising article to several subscales of a well-known instrument. Authors of the JA article decided not to measure the individual facets and, instead, selected items from the subscales, added several more not in the subscales, and combined them with the implication that the score measured the construct championed by the cited author(s). The acceptable alpha they reported for their scale probably reinforced their belief that the items adequately represented one construct rather than multiple ones. No evidence was provided that the set was unidimensional.
3. From the Journal of Retailing, an article's authors stressed that they only used established and validated scales. Wonderful; but wait . . . one of the scales they used had six items but only two of them were from the scale in the cited source. Not only did the authors imply that the other four items came from that scale but they did not provide evidence that the six items measured the same construct. Thus, unlike what the authors claimed, the scale they created was not from the cited source and was not validated as they claimed.
4. The authors of an article in the Journal of Consumer Research cited a well-known instrument and then "adapted" it by combining items from three of that instrument scales even though the developer of the instrument showed that those items loaded on different factors. The authors of the JCR article did not justify the combination of the items nor did they provide evidence that the scale they created was unidimensional.
5. In a Journal of Marketing article, the authors wanted to use an abbreviated version of a scale to measure a construct. They cited an article in which rigorous examination of an abbreviated version of that construct was proposed, tested, and recommended. The authors of the JM article used a different set of items from the full version of the scale, did not provide evidence of their scale's unidimensionality and, in fact, the scale's alpha was lower than what is typically considered to be acceptable for theory testing.
Do you see a pattern here? There are some researchers who think is it acceptable to take items from multiple scales that measure different constructs and treat them as a unidimensional scale. It is improper when authors do not justify what they have done. It is even worse when they do not provide evidence that the items form a unidimensional measure. Worst of all are authors who say they have used "validated" scales yet the items have been taken out of the "validated" sets, combined with other items, and no evidence of their unidimensionality is provided.
In conclusion, creating scales without justification by selecting items from measures of different constructs and without empirical evidence of their unidimensionality is unacceptable. Justification of what is being done and providing empirical support for it should be a requirement of a paper's acceptance at our discipline's top journals. Why aren't these standards required and enforced today?
Related Reading
Bruner II, Gordon C. (1998), "Standardization & Justification: Do Aad Scales Measure Up?" Journal of Current Issues & Research in Advertising, 20 (Spring), 1-18.
Bruner II, Gordon C. (2001), "Beyond Validity," Office of Scale Research, Technical Report #0101. https://scaleresearch.siu.edu/reports/tr0101.pdf
Bruner II, Gordon C. (2012), "Ignoring Dimensionality," Scale-related Pet-Peeves, Blog #22. https://scaleresearch.siu.edu/petpeeve22.html
Bruner II, Gordon C. (2015), "Questionable Scale Unidimensionality," Scale-related Pet-Peeves, Blog #28. https://scaleresearch.siu.edu/petpeeve28.html