Generalization

Generalization is the claim that a finding holds beyond the particular people, setting, and operations that produced it. External validity is the name of the argument for that claim (Shadish, Cook, & Campbell, 2002). It is not a bonus point added to a study that already has internal validity. It is a different inference, and it needs its own reasons. A study can show, cleanly, that this manipulation changed this measure in this sample, and say nothing reliable about other samples. Internal validity does not imply external validity. External validity does not repair a confound. The two arguments are checked separately, and a paper that wins one is not entitled to speak as if it had won both.

Cozby and Bates ask three generalization questions that keep the claim from staying fuzzy (Cozby & Bates, 2020). Can the result be generalized to other people? To other settings? To other ways of operationalizing the variables? A failure on any one of them is a boundary, not a shame. Boundaries are how a literature becomes accurate. “This held for these people, in this setting, with these operations” is a scientific sentence. “This is how humans work” is a different sentence, and it has to be earned three times.

The sampling chapter already warned that convenience samples of university students are a particular slice, and Henrich, Heine, and Norenzayan’s review is the citation for how unusual that slice can be (Henrich, Heine, & Norenzayan, 2010). Generalization to other people is not accomplished by adjectives. It is accomplished by sampling those people, or by a series of studies that vary the population on purpose and watch the effect. A discussion section that “suggests the finding may generalize” without a reason is a wish. A reason would be a feature of the procedure that should not depend on the campus, plus at least one replication off the campus.

Mundane realism is the extent to which the surface events of a study resemble ordinary life. Experimental realism is the extent to which the situation engages participants so that they respond as they would to a real event of that psychological kind, whether or not the surface looks ordinary. A dull questionnaire about an emergency can have mundane content and no engagement. A stark laboratory task can engage the process the theory is about. External validity cares more about the process than about the furniture. A study that looks realistic and fails to engage the process does not generalize to the life it resembles. It generalizes to people filling out a dull form.

A direct replication repeats a study as closely as possible: same operations, same kind of sample, same setting. It asks whether the finding was a one-lab accident. A conceptual replication changes the operations or the setting on purpose, in a way the claim says should not matter, or should matter in a predicted way. It asks about the reach of the construct rather than about the particular script. Both are generalization tools. A literature that contains only the original demonstration contains no generalization evidence yet, however confident the original paper sounded. Failed replications are part of the evidence. A file that publishes the successes and forgets the failures overstates the reach.

A field experiment assigns a treatment in an ordinary setting: a classroom, a workplace, a public program. It often gains mundane realism and a population the laboratory does not have, and it often loses the tight control that made internal validity easy. Confederate behavior varies. People talk to each other. Attrition is messier. The trade can be worth it when the question is about that setting. It is not a moral upgrade over the laboratory. A confounded field experiment is still confounded, and the ordinary lighting does not fix it. The report should say which validity was purchased and which was spent, using the same honesty the control chapter asked for indoors.

An interaction, from the factorial chapter, is sometimes bad news for a careless generalization. If the treatment helps novices and does nothing for experts, there is no single effect to export. The effect is conditional. Reporting the interaction is how generalization becomes accurate rather than broad. A main effect that averages over a reversal should not be the sentence that travels into a textbook. The conditional sentence should travel. This is the same discipline as effect size. A number, or a phrase, that erases the condition of the effect is a larger claim than the cells supported.

Time is a setting. A finding from one decade, one platform, or one legal environment can be historically local. Archival and survey results are especially exposed, because the records and the norms move. A replication a decade later is not a pedantic extra. It is the check on whether the setting was part of the cause. Laboratory processes can be historically local too, when the manipulation depends on a belief or a technology that participants no longer hold. Experimental realism has to be rebuilt when the culture that made the staging believable has changed.

Diversity of samples is a methods requirement, not a slogan. A series of studies that varies age, language, and material conditions, and that reports where the effect shrinks, is a generalization program. A series that repeats the same pool and cites the repetition as robustness has replicated reliability inside one pool. Both activities are useful. Only the first answers the first row of Figure 20.1. The paper’s limitation paragraph is where this honesty either happens or is replaced by a ritual sentence no one is meant to use.

This course has followed a claim from a question, through ethics, constructs, operations, variables, measurement, sampling, observation, surveys, experiments, factorials, single-case and quasi-experimental comparisons, description, and inference, and landed on the question of how far the claim may travel. The through-line is the match between sentence and design. Description, prediction, cause, and explanation are different goals. Reliability and validity are different virtues. Assignment is not sampling. A p value is not a size, and a size in one sample is not yet a fact about other people. A reader who can apply those distinctions to a new paper has the methods course in hand.

The next course on the map is Statistics for Psychology. It is where the descriptions in the previous chapter and the tests in the inference chapter get their machinery: distributions, standard errors, power calculations, ANOVA for the factorial designs, and regression for the associations. Later advanced courses take up multivariate models, psychometrics, and structural equation models for researchers who will build measures and multi-variable explanations professionally. Those courses do not replace this one. They assume it. A sophisticated model on a confounded design, a convenience sample narrated as a population, or a scale that was never checked for what it measures, is a precise treatment of a methods mistake.

The area courses, biological through history and systems, are where these methods get used on particular literatures. When you go there, take the questions from this course with you. Who was studied, what was actually manipulated or only measured, what else traveled with the cause, how large was the effect, and which people and settings are still untested? A literature that cannot answer those questions is not yet ready for a grand sentence, however familiar the grand sentence has become. PSY 200 stops at that standard. The rest of the map is the chance to apply it.