Reliability

Reliability is the consistency of a measure. A reliable procedure gives a similar result when the thing being measured has not changed: the same person on two close occasions, the same behavior coded by two careful observers, the items inside one scale agreeing with each other. Reliability is a property of the scores a procedure produces in a stated use, not a medal engraved on the questionnaire. A scale that is reliable in one language, one population, and one setting can fall apart in another. The paper owes the reliability of the scores it actually analyzed.

The useful picture is an equation methods courses teach early. An observed score is a true score plus measurement error. The true score is the stable standing the procedure is trying to capture. The error is everything else that moved the number: a misunderstood word, a loud hallway, a coder’s bad day, a guess on an item. If error is large relative to the true differences among people, the ranking of people jiggles from one occasion to the next, and any correlation that uses the score is dragged toward zero. An unreliable measure does not create phantom effects so much as it hides real ones and makes group differences harder to see. That is already a reason to care, before validity even enters.

Reliability is not validity. A procedure can be perfectly consistent and consistently wrong. A scale that always adds ten points, or a clock that is always five minutes fast, is reliable. It is not accurate. Validity, the next chapter, asks whether the score means what the researcher claims. Reliability is the prior question: whether the score means the same thing twice. A measure that does not agree with itself cannot be shown to agree with the construct. The reverse does not hold. Agreement with itself does not prove the construct.

Test-retest reliability is the correlation between scores from the same people on two administrations, separated by an interval in which the construct should not have moved. A short interval lets memory of the first answers inflate the correlation. A long interval lets real change look like error. The interval has to match the claim. A mood measure is not expected to be stable across a month. A trait measure that collapses across a week is not measuring a trait, whatever the title says. The coefficient is a correlation: it describes how well the ordering of people is preserved. It does not say the scores were identical, and it does not say the test is fair.

Internal consistency asks whether the items on a scale are pulling in the same direction, as they should if they are indicators of one construct. The common index is Cronbach’s alpha, which rises when items covary and when there are more of them (Cronbach, 1951). Alpha is not a test of unidimensionality by magic. A scale with two clusters of items can post a respectable alpha while measuring two things. Nor is there a law that alpha must exceed a particular cutoff. Researchers often treat values around .70 as a rough lower bound for a scale still under development and look for higher values when a score will be used to make a decision about a person. The convention is a warning light, not a proof. A paper should report the alpha of its own scores and say what the items were.

Split-half reliability is an older cousin: the items are divided in two, the halves are correlated, and the correlation is adjusted because each half is shorter than the full scale. Alpha generalizes that idea across the ways of splitting. Interrater reliability is the agreement between observers who code the same behavior. For categories, a raw percent agreement is inflated by chance, especially when one code is common. Cohen’s kappa corrects for the agreement that chance would produce given the raters’ own tendencies to use each code. Two raters who both mark “nothing happened” on almost every interval will agree often and may still have a modest kappa. The correction is the point. A coding scheme that cannot survive a second rater is a private impression with a spreadsheet.

The remedies live in the procedure, not in the discussion section. More items that actually tap the same construct tend to raise internal consistency, because random error on single items averages out. Clearer anchors on a rating scale, a practiced coding manual, and a pilot in which raters discover where they disagree all reduce error before the real sample is spent. Throwing out items that do not correlate with the rest can raise alpha, and it can also quietly change the construct. The remaining items may be easier, narrower, or more socially safe. Reliability purchased by emptying the scale of its difficult content is a validity problem wearing a reliability costume.

Reliability also limits the size of the associations a study can find. If the outcome is mostly error, the correlation with a predictor is attenuated. A paper that reports a small relationship and a low alpha has not shown that the constructs are unrelated. It has shown that these scores barely held still. The honest sentence says both things. The cure is a better measure or more measurement, not a larger adjective in the conclusion.

Report the estimate that matches the claim. A trait paper without a stability coefficient is asking the reader to assume the score would have been the same next week. An observational paper without interrater agreement is asking the reader to trust one coder’s eye. Those omissions are methods failures, not housekeeping. The next chapter asks the question reliability cannot answer: given a score that holds still, is it a score of the construct named in the title?