Validity, including construct validity

Validity is the degree to which the evidence supports the use of a score, or a design, for a stated conclusion. The word is not a property that a test owns in the abstract. A driving test can be valid for licensing and useless as a measure of courtesy. A depression scale can be valid for ranking symptom burden in one clinical setting and misleading as a single number in another language. Shadish, Cook, and Campbell organize the threats under four headings that methods courses now treat as the map: construct validity, internal validity, external validity, and statistical conclusion validity (Shadish, Cook, & Campbell, 2002). This chapter centers on construct validity, because that is the question of whether the operations match the idea, and it places the other three so they are not collapsed into one compliment.

Construct validity asks: do these operations measure or manipulate the construct named in the paper, rather than some neighbor of it? Internal validity asks: in this study, did the proposed cause produce the difference, or did something else that rode along with it? External validity asks: would the conclusion hold for other people, settings, and operations? Statistical conclusion validity asks: did the analysis support the claim that a relationship exists, at a size worth stating, or did noise, a violated assumption, or a fishing trip produce the sentence? A study can be strong on one and weak on another. The famous case is a tight laboratory experiment with excellent internal validity and a narrow claim on the world outside the room.

Cozby and Bates teach these distinctions so that a reader stops saying “the study is valid” as if that were one switch (Cozby & Bates, 2020). The switch is always “valid for what inference?” The rest of this chapter works the construct question in the pieces researchers actually collect: the content of the items, the pattern of correlations with other measures, and the criterion the score is supposed to forecast.

Face validity is the extent to which a measure looks, on its surface, as if it measures the construct. Face validity helps participants take the task seriously and helps a reviewer see the author’s intent. It is the weakest evidence of meaning. A measure can look right and tap something else, and a measure can look odd and work. Projective tasks and some reaction-time procedures have modest face validity and are defended, when they are defended, by other evidence. A paragraph that says only “the items appear to measure anxiety” has reported face validity and stopped before the argument.

Content validity is the extent to which the measure covers the domain of the construct and does not leave out a major part or smuggle in a foreign one. A course exam that never touches half the topics on the syllabus has a content problem even if the included items are clear. Content validity is judged against a definition of the domain, often by people who know the domain, and the judgment should be explicit. A researcher who writes ten items in an afternoon has not performed that judgment. The definition of the construct, from the constructs chapter, is what makes a content gap visible. Without the definition, every item looks relevant.

Convergent validity is evidence that the measure correlates with other measures of the same construct, or of constructs that the theory says should travel with it. Discriminant validity is evidence that the measure does not correlate as strongly with measures of different constructs, especially the lookalikes that would make the score ambiguous. A new scale of empathy that correlates .80 with a scale of social desirability and .20 with other empathy scales has failed the interesting test. It converges with the wish to look kind. Campbell and Fiske’s logic, which the validity tradition absorbed, is that a measure should agree with its kin more than with its neighbors. One correlation is a hint. A pattern is an argument.

Criterion validity asks whether the score relates to an outcome that the construct ought to relate to, an outcome that is not just another copy of the same questionnaire. Concurrent validity is the relationship with a criterion measured at the same time: a screening score alongside a clinical judgment made the same week. Predictive validity is the relationship with a criterion measured later: an admission test alongside grades the next year, a risk score alongside a later event. Predictive validity is the harder and often more useful claim, because the criterion was not sitting in the same session, available to color both scores.

The criterion has to be a real stake, and it has to be measured with its own integrity. Predicting a second form of the same test shows reliability more than it shows criterion validity. Predicting a supervisor’s rating shows something about the world only if the rating is not itself a rumor of the test score. When the criterion is rare, noisy, or biased, a modest correlation can be the honest ceiling. The paper should say what the criterion was and when it was collected. “The scale predicts important outcomes” without the outcome, the interval, and the size of the relationship is an advertisement.

Construct validity of a manipulation is the same argument applied to an independent variable. If the researcher claims to have manipulated accountability, the study needs evidence that accountability changed and that neighboring states, such as mere anxiety or mere extra time, did not change in a way that could carry the result. A manipulation check is a measure inserted to show that the operation moved the construct it was supposed to move. A failed check does not prove the manipulation did nothing. It does remove the easy sentence. The discussion then has to say what was actually manipulated, which may be different from the word in the title.

Internal validity is the subject of the confound chapter, because its threats are design threats: something else changed with the cause. External validity is the subject of the generalization chapter, because its question is about other people and other settings. Statistical conclusion validity returns when this course turns to description and inference. Parking them here would turn one chapter into the whole book. Naming them here keeps a reader from using “valid” as a single blessing.

A practical order of attack, when a new measure appears in a paper, is short. First reliability: did the score hold still? Then content: what domain was promised and what was written? Then the correlation pattern: kin versus neighbors. Then the criterion, with its date. Then the manipulation check, if the variable was a cause rather than a measure. A measure that skips these and proceeds directly to a bold conclusion is asking for trust the evidence has not built. The next chapter is the one that asks whether the difference between conditions was the difference the author named, or a companion that was never introduced.