Experimental design

An experiment manipulates an independent variable, assigns people to its levels by a procedure that does not let their preexisting traits choose the level, and measures a dependent variable. The assignment is what distinguishes an experiment from a comparison of groups that already existed. Random assignment, introduced with confounds, is the usual procedure: each person has the same chance of each condition, so that the crowd of unmeasured participant variables is scattered rather than aligned with the treatment. When assignment is truly random and the conditions differ only in the intended ingredient, the three conditions for a causal claim are finally in one design. Covariation is the difference in the outcome. Temporal precedence is guaranteed because the researcher set the cause first. Alternatives are balanced by the assignment and by control of the procedure.

That sentence is an ideal the procedure has to live up to. A “randomized” study in which the experimenter steers difficult participants into the easier condition is not randomized. A randomized study in which the treatment condition also gets a warmer experimenter is randomized and still confounded. The word experiment in a paper is a claim about assignment and about what else was held still. The reader checks both, in the methods, before accepting the verb in the abstract.

Random sampling and random assignment remain different acts. Sampling chooses who enters the study. Assignment chooses which condition they enter. An experiment on a convenience sample can have strong internal validity and weak external validity at the same time. That combination is common and it is not a scandal if the paper’s sentence matches it. The scandal is a causal claim about “people” from a design that only earned a causal claim about the people who were assigned.

In an independent-groups design, also called a between-subjects design, different people serve in different conditions. Each person is assigned to one level. The comparison is between people. The advantage is cleanliness of experience: a person in the treatment has not also lived through the control, so order, practice, and contrast cannot explain the difference. The cost is people. Differences among people become part of the error, and the study needs enough people in each condition for the treatment difference to show through that noise.

The posttest-only design assigns people, administers the treatment or the control, and measures the outcome once. It is the basic experiment. It relies on random assignment to make the groups comparable at the start, which is why a check on that comparability matters when samples are small: a fluke imbalance on a variable that affects the outcome is still possible. The pretest-posttest design measures the outcome before the treatment as well as after. The pretest can show that the groups started together, can increase the sensitivity of the analysis by accounting for each person’s starting point, and can answer whether anyone changed rather than only whether the groups ended apart. The pretest can also tip participants off, create a testing effect, or be impossible when the measure itself would ruin the manipulation. The pretest is a tool with a price, not a moral improvement.

A matched-pairs design forms pairs of people who are similar on a variable that strongly predicts the outcome, then assigns one member of each pair to each condition. Matching is an attempt to buy some of the sensitivity of a within-person comparison without giving every person every condition. The match is only as good as the matching variable. Matching on a variable that barely predicts the outcome wastes the effort. Matching does not replace random assignment inside the pair. If the researcher always puts the slightly more eager member into the treatment, the match has been undone by hand.

In a repeated-measures design, also called a within-subjects design, each person serves in every condition. The comparison is inside the person. The statistical advantage is large: stable differences between people are subtracted out, so a smaller sample can detect a difference, and the researcher knows the same person experienced both levels. The design threat is order effects. Practice makes the second task easier. Fatigue makes the second task worse. Carryover means the first condition changes the person in a way that is still present during the second: a mood induction that has not worn off, a strategy that cannot be unlearned, a drug still in the body.

Counterbalancing, already named, is the standard answer when order effects are symmetrical enough to be balanced. Complete counterbalancing uses every possible order. When the number of conditions makes that impossible, a Latin square and related plans give each condition an equal turn in each position. Counterbalancing does not remove an irreversible carryover. If the first condition teaches a skill that cannot be forgotten, the second condition is always taken by a person who has the skill, no matter the order. In that case the repeated-measures design is the wrong design. Independent groups are slower and cleaner.

A within-subjects design also changes the psychology of the participant. Someone who sees both the flattering and the unflattering version of a message may guess the hypothesis. That guess is a demand characteristic, treated in the next chapter. Sometimes the hypothesis is obvious only because both levels were shown. Between-subjects designs hide the other level. The choice of design is therefore not only a statistical preference. It is a decision about what the participant is in a position to notice.

The hypothesis decides. If the question is what this person does under two conditions they might meet in sequence, a repeated-measures design resembles the world being studied, and order effects may even be part of the phenomenon. If the question is the effect of a lesson that cannot be unlearned, independent groups are the design that keeps the untaught comparison alive. If a strong predictor of the outcome is known and cheap to measure, matching or a pretest can spend that information instead of leaving it in the error. These are engineering choices. They are not a ladder of prestige.

A reader reports the design in one accurate sentence: who was assigned, to what, whether anyone experienced more than one level, and what the control contained. “A randomized controlled experiment” is a category. The sentence that distinguishes a posttest-only design from a pretest design, and a between-subjects design from a within-subjects design, is the sentence a methods course exists to produce. The next chapter stays inside the experiment and asks how the manipulation and the measure are actually built so that the design on the page survives contact with a room full of people.