Conducting experiments

A design on paper becomes an experiment when a room, a script, and a measure make the independent variable the only systematic difference. Straightforward manipulations change the situation by instruction or by stimulus: a written warning, a time limit, a film, a dose, a piece of feedback that is in fact the same for everyone inside a condition. Staged manipulations construct an event the participant believes is real: a confederate who pretends to be another participant, a false score, an emergency that is acted. Staged manipulations can produce strong situations. They also raise the ethics of deception and the risk that the staging, rather than the named variable, is what people respond to. The actor’s warmth can become the confound.

The strength of the manipulation has to be enough to move the construct and not so theatrical that it changes a dozen other things. A whisper of a manipulation produces a null result that is about the whisper. A manipulation that humiliates people may “work” and fail ethics, or may work by humiliation rather than by the theory’s mechanism. Pilot work is how strength is chosen: a small run whose job is to see whether people notice the manipulation, believe the staging, and can complete the measure. A pilot is not a sneak preview of the result to be added into the main sample. It is a test of the apparatus.

A manipulation check, defined with validity, belongs in the session when the claim depends on a particular construct having moved. If the check fails, the discussion does not get to keep the construct’s name. It reports what the operation did. Some manipulations are so direct that a check is redundant: the time limit either was four minutes or it was not. The check earns its place when the operation is a psychological state that might not have arrived.

Dependent variables in experiments are usually one of three kinds. Self-report asks the person what they think, feel, or would do. It reaches constructs that behavior will not show in a twenty-minute session, and it inherits every survey problem from the previous chapter. Behavioral measures record what the person does: whether they help, how long they persist, what they choose when a real cost is attached. They are harder to dismiss as talk, and they can be narrow. Helping in one staged emergency is not helping in a life. Physiological measures record a bodily signal: heart rate, skin conductance, a brain measure. They feel objective. They are still operational definitions with a construct problem. Arousal is not an emotion. A brain signal is not a thought. The inference from the signal to the construct has to be argued, not assumed because the device was expensive.

Sensitivity is the ability of the measure to detect a difference if one is there. A ceiling effect occurs when most people score at the top, so the treatment has nowhere to go. A floor effect occurs when most people score at the bottom. Both produce null results that are about the ruler. A pilot that looks at the distribution, not only at the mean, catches them. So does a measure with enough range, and items that are not so easy or so hard that everyone lands in the same bin. Reliability matters here for the reason already given: an error-soaked outcome hides the manipulation.

Multiple measures can strengthen a result when they converge, and they can tempt a researcher to report the one that moved. The list of dependent variables is part of the design and should be the list in the report. Dropping a measure because it was inconvenient is an inference problem, saved for the chapter on claims. The conducting problem is simpler. Decide the measures because they match the construct, check that they can vary, and then leave the list alone.

Demand characteristics are cues that tell a participant what the hypothesis is and what a good participant would do. A leading questionnaire, a title on the consent form, a confederate who overacts, and a within-subjects design that displays both conditions can all supply the demand. Participants may then confirm the hypothesis, or rebel against it, or try to look normal. None of those behaviors is the construct. Reducing demand means a cover story when ethics allow, a measure that is not a transparent ballot on the hypothesis, and a debriefing that asks what people thought the study was about. If most of them name the hypothesis, the results are ambiguous in a specific way.

Experimenter expectancy is the researcher’s own hypothesis leaking into the procedure: a slightly longer pause, a warmer tone, a coding decision in the predicted direction. Rosenthal’s work on experimenter effects showed that expectations can be transmitted in behavioral research and can shape what is recorded (Rosenthal, 1966). The practical controls are a script, training, and blinding. A single-blind procedure keeps the participant from knowing the condition or the hypothesis. A double-blind procedure keeps the person who runs the session and, where possible, the person who codes the outcome from knowing the condition as well. Blinding is not always possible. A surgeon knows which operation was done. When blinding is impossible, the paper says so, and automated delivery of instructions becomes attractive because a screen does not have an encouraging eyebrow.

A placebo control gives the comparison group the form of the treatment without the ingredient, so that expectation is matched. Placebo control is essential when the claim is about an ingredient beyond expectation, and it is a deception that the ethics chapter already constrained. A study with no placebo that claims a specific mechanism over and above hope has not run the comparison its verb requires. A study whose honest question is whether the whole package, hope included, helps people may not need a placebo. It needs a sentence that does not overclaim the ingredient.

Conducting includes the dull protections that keep a data file meaningful. Conditions are labeled in the data by the assignment, not by the researcher’s later impression of who “really” received the treatment. People who break off are counted, because attrition that differs by condition is a threat already named. Deviations from the script are noted rather than remembered. A pilot study is the place to discover that the script is too long, the cover story is unbelievable, or the measure is at ceiling. The main study is the place to run the repaired procedure on the sample that will be reported, without folding the pilot in to enlarge the effect.

The end of the session returns to ethics. Debriefing is part of conducting, not an optional courtesy, when any deception or stress was involved. It is also a source of methods information: what people noticed, what they found impossible, what they did not believe. A researcher who does not listen to the debriefing will repeat a broken staging and call it a replication. The data then leave the room and enter description and inference, which are later chapters. Nothing in those chapters can repair a manipulation that was confounded by the experimenter’s smile. That repair had to happen here, before the first real participant, or be confessed in the report if it did not.