Main Content Area
Scale-related Pet-Peeves
Blog #24
Setting Standards for Scale Usage and Reporting
In my previous posting, I expressed deep concern about the unevenness of scale descriptions in articles published in top marketing journals as well as the variance in scale quality. I challenged journals to increase their standards with respect to measurement. In this blog I want to take the next step and propose some standards I believe should be followed.
Minimal Standards
Multi-item scales should be used when measuring psychological constructs. If not, explain why not. There may be justifiable reasons (e.g., Diamantopoulos et al. 2012) but they deserve to be explained rather than glossed over as if it does not matter.
When multi-item scales are used, the following information is required either in the main body of the paper or its appendices:
- Description of the scale's origin: Merely stating that a scale was used without specifying its source is unacceptable. Period. If it is original, authors should explain why previously published measures of the construct were not used. There may have been no known measure of the construct or the known measures were not acceptable in some way. If a scale is original, more information may be required than if it has had a history of usage and known psychometric quality. Authors should not describe a scale as adapted or something to that affect (borrowed, modified) unless their measure is a slight modification of something that has been used previously by someone else. As I have lamented in previous blogs, it is very common for the authors to use the term "adapted" too loosely. Countless times I have found a so-called "adapted" scale had little in common with the measure in the cited article. Even worse is when authors simply give a cite without any other description and leave it to the reader to assume they borrowed the scale in tact from the cited source. In reality, the authors of the paper under review may have developed the scale based on concepts found in the cited article rather than borrowing a scale created by the cited author(s). Most disturbing of all is when no cite is given and yet the scale was taken from another source.
- The authors should provide some evidence of unidimensionality. This will typically involve some form of factor analysis.
- The authors should give evidence of a scale's reliability. In most cases, that will be a measure of internal consistency such as Cronbach s alpha or composite reliability. In some cases, another form of reliability is called for: temporal stability (aka test-retest). This is particularly appropriate when the construct being measured is supposed to remain stable over time such as with a personality trait.
- If translation from one language to another has been conducted in the development of a questionnaire and its scales, that process should be described.
Increasing Standards
Doing those things might be considered the minimum that authors should provide and, believe me, they are not followed nearly enough even in our premier marketing journals. However, higher standards should be considered at the best journals or at any scholarly journal when a paper is submitted which has the main purpose of presenting a new scale. The following are further standards that should be considered by a journal beyond the minimum listed above:
- Provide multiple forms of evidence in support of the scale's validity. There are several different forms of validity though I am not advocating all forms must be addressed in any one study.
- If the scale is used in a cross-national study then even more evidence is expected, e.g., configural invariance, scalar, and metric.
Nothing I have said here may be new. The problem may not be knowing these goals but, instead, using them consistently. The bottom line is that whatever they are, standards should be set by journal boards and then stated clearly at a journal s website. Once authors and reviewers are aware of them, I would hope that the quality of the scales used and the clarity with which they are reported will improve much beyond what it is now. Subsequently, our confidence in a study's results and authors conclusions may improve as well.
Diamantopoulos, Adamantios, Marko Sarstedt, Christoph Fuchs, Petra Wilczynski, and Sebastian Kaiser (2012), "Guidelines for Choosing between Multi-item and Single-item for Construct Measurement: A Predictive Validity Perspective," Journal of the Academy Science, 40 (3), 434-449.