Confounds and experimental control

A confound is a variable that changes along with the independent variable and that could itself have produced the change in the dependent variable. The two conditions differ in the way the researcher intended, and they also differ in some other way that the design did not separate. The result is ambiguous. The data show a difference. They do not show which of the traveling variables was responsible. Internal validity, named in the previous chapter, is the name for the argument that this ambiguity has been closed (Shadish, Cook, & Campbell, 2002).

Confounds are easy to miss because the intended variable is the one in the title. A “meditation” condition that also includes a kind instructor, a quieter room, and twenty extra minutes is not a manipulation of meditation alone. Any of those companions could carry a later difference in reported calm. The companions are not nitpicks. They are alternative explanations, the third condition of a causal claim from the opening chapter. Until they are blocked, held constant, or measured and dealt with, the causal verb is early.

A confound is not the same thing as a nuisance that adds noise. Noise makes scores wobble and makes effects harder to detect. A confound systematically favors one condition. Adding participants reduces noise. Adding participants does not repair a confound, because every new participant inherits the same tangled comparison. The repair is structural. The conditions have to be separated so that the intended variable is the one that differs.

Campbell and Stanley catalogued the classic threats to internal validity, and the catalog is still the one methods books teach (Campbell & Stanley, 1963). History is an event besides the treatment that occurs between a pretest and a posttest. A campus program evaluated across a semester in which a public crisis also occurred cannot give the program the whole credit. Maturation is change that would have happened anyway because people grow, heal, tire, or get bored. Children tested twice will often look “improved” because they are older. A comparison that has no untreated group from the same stretch of time cannot tell maturation from treatment.

Testing is the effect of taking the measure the first time. People improve on a puzzle because they saw the puzzle, not because of the lesson that came between. Instrumentation is a change in the measuring device itself: a rater who becomes harsher, a machine that drifts, a questionnaire revised halfway through. Regression to the mean is the statistical tendency of extreme scores to be less extreme on a second measurement, because extreme scores were partly error. Selecting the most distressed people and then rejoicing that they are less distressed after a program is a design that regression will flatter even if the program is empty.

Selection is a preexisting difference between the groups being compared. It is the central threat whenever people are not assigned to conditions. The people who choose the morning class, the treatment, or the survey are not a coin-flip version of the people who do not. Attrition, also called mortality in the older lists, is loss of participants that differs by condition. If the most distressed people quit the treatment group and stay in the comparison group, the treatment will look helpful because of who remains. An interaction of threats is possible and ordinary: selection plus maturation, when the groups were already on different slopes before anyone intervened.

Experimental control is the set of procedures that keep rivals from favoring one condition. The most important, in a true experiment, is random assignment: each person has the same chance of landing in each condition, so that preexisting differences are tossed into the conditions rather than lined up with them. Random assignment is not random sampling. Sampling, the next chapter, is about how people get into the study. Assignment is about how, once they are in, they get into conditions. A convenience sample can still be randomly assigned. The causal contrast inside the sample can be clean while the road to other populations stays closed.

Holding a variable constant is control by removal. If both conditions use the same room, the same duration, and the same experimenter reading the same script, those variables cannot explain a difference. The cost, as the variables chapter said, is that the study no longer speaks about other rooms and other experimenters. Counterbalancing is control used when every person receives more than one condition and the order would otherwise be a confound. Half the people get A then B. Half get B then A. Order still happens. It no longer belongs to only one condition. Blinding, taken up again when experiments are conducted, keeps expectations from becoming an extra manipulation.

A control condition is the comparison that makes the independent variable visible. “Control” does not always mean “nothing.” It means the condition that has everything the treatment has except the proposed active ingredient. A placebo is a control for expectation. An alternative-task control is a control for time and attention. A no-contact group is a weak control if the treatment group also received attention, because attention is then a confound. The right control is the one that isolates the ingredient named in the hypothesis. A vague “we had a control group” does not say which rival was answered.

The reader’s question is concrete. List what differed between the conditions besides the named cause. If the list is empty or balanced, internal validity is in good shape and the remaining arguments are about measurement and generalization. If the list contains a plausible cause of the outcome, the paper’s verb is too strong. Plausible means a variable that could actually move this dependent variable, not a philosophical worry. A difference in the font of a slide is rarely a plausible cause of a health outcome. A difference in how hopeful the recruiter sounded is often a plausible cause of whether people return for session two.

Some rivals can be measured even when they cannot be assigned. Measuring them does not equal random assignment. It lets a later analysis ask whether the treatment difference shrinks when the rival is held statistically constant. That move is an argument, and it is weaker than a design in which the rival never varied, because the measurement of the rival has its own error and because unmeasured rivals remain. The chapter on quasi-experiments returns to this weakness. For now the rule is blunt. The cleanest control is a design in which the rival does not get a chance to travel with the cause.

None of this makes experiments the only science. Description and prediction do not need this chapter’s full apparatus, and they must not pretend they have it. When a paper uses causal verbs about a measured predictor, the threats in Figure 9.1 are an open list, with selection at the top. The next chapter turns from how people are placed into conditions to how they got into the study at all.