Asking people about themselves: surveys

A survey asks people to report on themselves: attitudes, facts about their lives, frequencies of behavior, intentions. The appeal is obvious. Many of the constructs this field cares about are not visible from across a room, and a well-built questionnaire can reach a probability sample that an hour in the laboratory never will. The risk is equally obvious. The answer is a behavior of its own, produced under the social conditions of being asked. Cozby and Bates treat question writing as a methods skill because a biased question manufactures the distribution the researcher then “discovers” (Cozby & Bates, 2020).

Three rules remove the most common wrecks. Ask one thing at a time. A double-barreled question ties two claims with “and” and then offers one yes: “Do you support more funding and stricter rules?” A yes may mean either, or both, and the researcher cannot tell. Use words the respondent uses. A technical noun that the researcher finds precise is a source of noise if half the sample guesses. Avoid a loaded question that builds the desired answer into the premise: “Do you agree that this harmful policy should end?” The word “harmful” did the work before the respondent arrived. Negative wording piled on negative wording (“Do you disagree that it is unwise…”) produces error that looks like a personality.

Closed questions offer fixed responses and are easier to count, compare, and ask of a large sample. Open questions let the person answer in their own words and are how a researcher learns the categories they would otherwise have invented badly. A serious survey often uses open questions in a pilot to build the closed options, then closed questions in the study that will be tallied. Skipping the pilot means the options are the researcher’s stereotypes, and respondents who do not find themselves in the list will choose the least-wrong box.

A rating scale asks for a degree rather than a yes. A Likert scale offers a symmetric run of agreement, from strong disagreement to strong agreement, around an item that states an attitude. The points should be labeled, or at least anchored at the ends, so that “4” means the same thing to different people. A semantic differential places the object between bipolar adjectives: cold to warm, weak to strong. Both are measures, with all the reliability duties of the reliability chapter. A single item is a thin measure. A scale of several items aimed at one construct can be checked for internal consistency. A pile of items about different topics is not a scale, even if they share a response format.

Question order changes answers. An earlier item can define a word, make a norm salient, or commit the person to a consistency they then protect. Asking about a specific scandal before asking about trust in an institution is a different survey from the reverse order. When order effects are a threat, researchers split the sample and give different orders, or they place the sensitive general question before the vivid specific one. Pretending order does not matter is a choice to leave a confound unmeasured.

Response sets are habits of answering that are not about the content. Acquiescence, or yea-saying, is the tendency to agree. A scale whose every item is worded so that agreement means a high score will confuse acquiescence with the construct. Including some reversed items helps, and reversed items bring their own risk if the negation is confusing. Social desirability is the tendency to give the answer that looks good. It is strongest when the behavior is admired or shameful and when the respondent is identifiable. Anonymity reduces it and does not abolish it, because people also manage the impression they make to themselves.

A survey inherits the sampling chapter whole. A brilliantly worded questionnaire given to a convenience sample supports conclusions about the people who answered, not about a population that was never framed. Nonresponse is especially poisonous in surveys because the topic of the survey is often the reason people refuse. A survey of drinking that drinkers decline will underestimate drinking. The achieved sample, the response rate, and a comparison of who refused, when any comparison is possible, belong in the results, not in an apology.

Interviews and questionnaires are different instruments. A questionnaire is filled out by the respondent, on paper or a screen, and is cheaper and easier to keep anonymous. An interview is spoken, and a skilled interviewer can clarify and can keep a respondent through a long schedule. The interviewer is also a situational variable. Tone, gender, haste, and the urge to help a struggling respondent all change answers. Interviewer training and a script are reliability procedures. A freewheeling “conversation about the topic” is not an interview schedule, and two interviewers will not have conducted the same study.

Panel surveys ask the same people more than once. They can establish temporal precedence that a single sitting cannot: the attitude this month, the behavior next month. They pay for that with attrition and with testing effects. People who remain in a panel are more dutiful than people who leave, and people who have answered once have been prompted to think about the topic. Those are the threats of testing and attrition from the confound chapter, now inside a questionnaire. Naming the design “longitudinal” does not retire them.

A cross-sectional survey, everyone measured once, is a machine for description and for prediction. It can say how common a report is, and which reports travel together. It cannot, by itself, say which variable produced which, because temporal precedence is missing and selection is the whole design. Attitudes and behaviors measured on the same afternoon can be arranged into a causal story in either direction, and both arrangements will fit the correlations. The paper may prefer one story. The data have not voted.

The standard of a finished survey method is dull and sufficient. The items are available, the response options are defined, the order is stated, the sampling frame and the response rate are stated, and the reliability of any multi-item score is reported. A reader can then decide whether the distribution is a fact about the construct or a fact about the asking. Experiments, which the following chapters take up, buy causal leverage by assigning conditions. They often pay with a narrower sample and a shorter slice of behavior. Surveys buy reach and self-report. Knowing which currency a paper spent is the skill this chapter was for.