Quasi-experiments

A quasi-experiment studies a treatment or a policy when the researcher cannot randomly assign people to receive it. The program is offered to one school and not another. A law changes on a date. A clinic adopts a protocol for the patients who enroll. The independent variable is real in the world and messy in the design, because selection is no longer scattered by a coin flip. Shadish, Cook, and Campbell built the modern map of these designs precisely so that researchers would stop treating “we compared the group that got it with the group that did not” as if it were an experiment (Shadish, Cook, & Campbell, 2002). Cozby and Bates teach the same caution as part of a methods sequence (Cozby & Bates, 2020).

Quasi-experiments are not failed experiments. Many of the most important questions cannot be randomized: disasters, laws, illnesses that we will not assign, schools that will not flip a coin on a curriculum. The task is to build a comparison that answers as many of the internal-validity threats as the situation allows, and to match the verb to the threats that remain. A quasi-experiment with a good comparison can support a cautious causal argument. A quasi-experiment with no comparison supports a story about time.

The weakest common pattern is the one-group pretest-posttest design: measure, treat, measure again, and credit the treatment for the change. History, maturation, testing, instrumentation, and regression are all free to operate. The design is sometimes all an institution will allow. It is then a description of change, and the report should say so. Calling it evidence that the program works spends a word the design did not earn.

A nonequivalent control group design adds a comparison group that did not receive the treatment, measured on the same schedule. The groups are nonequivalent because they were not assigned. They may differ at the pretest, and those differences are the selection threat in numbers. A treatment effect is more credible when the groups were similar at pretest and diverge after the treatment, and less credible when they were already on different paths. Analyzing only the posttest ignores the information that would reveal the selection. Analyzing the change, and showing the pretests, is the minimum honesty.

An interrupted time series measures the outcome many times before and after a treatment that occurs at a known point: a law, a closure, a new rule. The comparison is the series with itself. A level shift or a slope change at the interruption is the evidence. A series that was already falling, and keeps falling at the same slope after the policy, is not evidence for the policy, even if the values after are lower than the values at the start. The picture of the whole series is the result. A two-point before-and-after sliced out of a long series is the weak design again, wearing a longer data set.

Combining the two ideas, a comparison series that did not receive the interruption, is stronger still. A policy in one region with a neighboring region measured the same way can answer the history threat that would have hit both. It cannot answer a history that hit only the treated region, and it cannot invent equivalence the regions do not have. Statistical adjustments, including matching people who look similar on measured covariates, try to shrink selection. They answer measured selection. They do not answer the variables nobody recorded. An adjustment is an argument. It is not random assignment, and the paper that calls it a “quasi-randomized experiment” is decorating a word.

Age cannot be assigned. Studies of development therefore use designs that this chapter’s logic covers, even when they are taught again in a developmental course for their content. A cross-sectional design compares people of different ages at one time. It is fast, and it confounds age with cohort: the era in which a person was born, with its schools, diets, and history. A difference between 20-year-olds and 70-year-olds in a single year is a difference between two generations, not a movie of one life. A longitudinal design follows the same people as they age. It unconfounds cohort from age for that cohort, and it pays with attrition, testing effects, and a long wait. A sequential design starts more than one cohort and follows each, so that age effects and cohort effects can be compared rather than swallowed together.

These are not decorations on a quasi-experiment chapter. They are the reason a methods student refuses the sentence “people become more this as they age” from a single cross-section. The sentence may be true. The design showed that two age groups differ now. The developmental psychology course on this map will use these designs to talk about particular findings. The method stays here: name the comparison, name the confound age always brings, and do not let the word development do the work of assignment.

Across this chapter the discipline is the same one as in the confound chapter, under harder conditions. List the rivals the design can answer. List the rivals it cannot. Put the causal verb only over the first list. Programs, laws, and lives are important enough to study with the designs we actually have. They are not so important that a weak design gets a strong verb as a courtesy.