Inferential claims without overclaiming

Inferential statistics use a sample to say something about a population, with an explicit allowance for sampling error. The machinery belongs to Statistics for Psychology. The manners belong here, because methods students meet p values long before they derive them, and the manners are where overclaiming happens. A test does not discover that a theory is true. It answers a narrow question about how unusual the data would be under a stated model of no effect, given the assumptions of the test. Cozby and Bates press this modesty because the literature is full of sentences larger than their tests (Cozby & Bates, 2020).

The null hypothesis is the specific claim the test puts under stress, usually that a difference or a correlation is zero in the population. The alternative hypothesis is the claim that it is not zero, or that it points in a stated direction. The test asks: if the null were true, and if the assumptions held, how often would a result at least this extreme appear? That frequency, under the null, is the p value. A small p value means the data are awkward for the null. It does not mean the probability that the null is true is small. It does not mean the effect is large. It does not mean the result will replicate. Those three misreadings are the overclaim this chapter exists to stop.

Alpha is the threshold the researcher chose, before seeing the result, for what will count as awkward enough. A conventional alpha is .05. The convention is a social contract, not a discovery about nature. A result with p = .04 and a result with p = .06 are nearly the same evidence. Treating them as opposite worlds, “an effect” and “no effect,” is a property of the threshold, not of the data. Reporting the p value and the effect size, rather than only whether a star was earned, keeps the continuity visible.

A Type I error rejects a null that is true. With alpha at .05 and a true null, a test of this kind rejects about five times in a hundred. A Type II error fails to reject a null that is false. Power is the probability of avoiding that second error, of detecting an effect of a stated size. Power rises with larger effects, larger samples, more reliable measures, and a less stringent alpha. It is planned, not discovered afterward. A small study that “found nothing” may have had almost no power to find the effect that would have mattered. The honest sentence is “we did not detect a difference, and the study was only sensitive to effects larger than this,” not “there is no difference.”

Statistical significance means only that the result crossed the chosen threshold under the null. Practical significance asks whether the size of the effect matters for the decision or the theory. A tiny improvement in a rating, significant in a sample of thousands, can be practically trivial. A large improvement that falls just short of .05 in a small careful study can be the more important lead. Effect size, from the description chapter, is how practical significance gets a number. Cohen’s conventions remain conventions (Cohen, 1988). The context of the outcome decides whether the number matters.

A confidence interval is a range of values for a population quantity, built so that the procedure which builds it catches the true value at a stated rate across repeated samples, often 95 percent. The interval is about the procedure’s long-run success, and it is also, in use, the set of population values that would not be embarrassing for these data. A confidence interval that runs from a trivial effect to a large one is a confession of imprecision. A paper that reports only “p < .05” and hides an interval from nearly zero to enormous has chosen the less informative sentence. Intervals put the size and the uncertainty in one object. That is why methods courses now ask for them beside, or instead of, a lonely threshold.

Multiple testing without a plan inflates Type I errors. Twenty tests at alpha .05 will, under a world of true nulls, produce about one rejection by chance. Choosing to feature that one rejection is not a discovery. It is the expected debris. A p-hacking pattern is the family of flexibilities that keep going until a threshold is crossed: dropping a condition, swapping an outcome, adding covariates after seeing the data, stopping the study when the p value looks good. Each flexibility can be innocent in isolation. Together they make the reported p value refer to a procedure that was not the procedure in the paper.

The corrections are design corrections. Register the outcomes and the comparisons before the data exist. Distinguish confirmatory tests from exploration, and label exploration as exploration. Report the tests that did not cross the threshold. Prefer an interval and an effect size so that a result just under the line and a result just over it look like neighbors. Replication, taken up in the last chapter, is the check that does not depend on the original team’s flexibility. None of these repairs requires the full theory of the test. They require telling the truth about how many looks produced the sentence.

Overclaiming also happens in ordinary verbs. “Proved,” “demonstrated the mechanism,” and “showed that X causes Y in people” are larger than a single p value even when the p value is tiny and the design is a real experiment. The experiment supports a causal contrast in the conditions that were run, in the sample that was assigned, on the measure that was analyzed, at a size the interval displays. The mechanism is a further claim. The population is a further claim. Statistics for Psychology will teach how the tests are built, how power is calculated, and how multivariate models and, later, structural equation models partition several variables at once. Those courses do not grant bigger verbs. They grant more precise ones. A methods reader who keeps the verb inside the design will not need a later course to undo a habit of overclaiming, which is the harder habit to undo.