Main Content Area
Scale-related Pet-Peeves
Blog #32
Quick & Dirty Scale Creation
It is happening too frequently! A researcher uses a scale with low face validity and the only evidence provided in support of the measure's appropriateness and quality is its level of internal consistency. What is going on? I continue to come across these bad practices by scholars who ought to know better and who are publishing their work in the highest of marketing-related journals.
Since I won’t name names here, let me merely say it is not uncommon in scholarly consumer research to see scales composed of items that represent multiple constructs. I am sorry, folks . . . that is unacceptable, especially when the authors cite information which indicates they are aware that the items come from scales measuring different constructs. When a scale’s face validity is poor, and no evidence is provided of the measure’s dimensionality, the paper should not be published by a top journal. As for me, my standards have risen over time and I am unlikely to put such a measure in a book of mine anymore.
Some would argue that researchers should not be expected to spend lots of time gathering and reporting evidence of a scale's quality, especially when other aspects of a study are considered more important. I understand that, at least, partially. What is stressed here is that there are some basics that should be done in every case and they can be done relatively quickly. The first thing one should do, especially when there is not much time to develop a scale, is select from among the scales that have already been developed for measuring the construct. Of course, the selected scale should also have high psychometric quality. If researchers cannot find a scale that meets their needs, they can create one IF THEY FOLLOW SOME VERY BASIC STEPS. Prior to data collection, testing face validity should play a prominent role in selecting items for the new scale. The efficacy of that, however, depends upon one’s expertise in the topical area. Once the data have been gathered, evidence of face validity and unidimensionality should be confirmed using factor analysis. That analysis should not only include the items proposed for the new scale but also have items that measure some other constructs. Finally, measurement of internal consistency should be conducted, such as with Cronbach's alpha. An important point here is that alpha does not take the place of evidence supporting face validity and unidimensionality.
A more difficult problem to deal with occurs when there is a clash between these basic indications of scale quality (face validity, alpha, and factor analysis). I hope to deal with the inconsistency of evidence in a future post. For the time-being, I urge researchers to realize that something is wrong when there is inconsistency and not to use the scale until it is resolved.
The bottom line is to try as much as possible to find a scale that has been used in past research and has strong evidence of high psychometric quality. But, if a new scale must be developed, take great care in borrowing items and/or creating them. Items should not be combined if they measure different constructs. Further, provide evidence of the scale's unidimensionality and discriminant validity by running a factor analysis that includes the items proposed for the new measure as well as items that measure different though related constructs. Quick doesn't have to be dirty!