Reliability vs validity: the difference, with examples

Every test result ends in a decision: pass or fail, hire or not, which class to join. Reliability and validity are the two questions that tell you whether that decision can be trusted. They are easy to mix up because both sound like "a good test", so this article takes them one at a time, with numbers you can check on your own exams.

The difference in one example

Picture a bathroom scale that always shows two kilograms more than your real weight. Step on it five times and it shows the same number five times: it is reliable. But the number is wrong, so using it to decide whether you lost weight for a medical check is not valid. Now picture a scale that is right on average but jumps by three kilograms each time you step on it. It is unreliable, and for that reason no single reading can be valid either.

Tests behave the same way. A vocabulary quiz used as a "reading comprehension" exam can give very stable scores and still measure the wrong thing. A well-aimed exam with only eight questions can measure the right thing, but so noisily that two equally able candidates end up on either side of the pass mark.

Reliability: how consistent are the scores?

No test measures perfectly. A candidate's score is their true level plus some error: the particular questions that happened to be chosen, a lucky guess, a tired afternoon, a strict or lenient marker. Reliability describes how much of the spread in scores comes from real differences between candidates rather than from that error. A reliability of 0.80 means, roughly, that 80% of the variation in scores reflects real differences and 20% is noise.

Ways to estimate it

• Internal consistency: do the questions in one test agree with each other? This needs only one sitting, which is why it is the most common figure. Cronbach's alpha is the general formula; KR-20 is the same idea for questions scored right or wrong, and for such questions it gives the same value as alpha.
• Test-retest: the same candidates take the same test twice, some time apart, and the two sets of scores are correlated.
• Parallel forms: candidates take two versions built to the same blueprint. This matters whenever you give different candidates different papers.
• Inter-rater: for essays and practical tasks, two markers score the same answers independently and their agreement is measured.

The standard error of measurement

A reliability coefficient is abstract. The standard error of measurement (SEM) turns it into points on your own scale:

SEM = SD × √(1 − reliability)

Say your exam scores have a standard deviation of 10 points and a reliability of 0.84. Then SEM = 10 × √0.16 = 4 points. A candidate who scored 62 has a true score that is, about two times in three, between 58 and 66, and about 19 times in 20 between 54 and 70. If the pass mark is 60, that candidate is not clearly above it. This is the practical reason reliability matters: it tells you how many candidates near the cut score could have landed on the other side by chance.

How much reliability is enough?

There is no single threshold, because it depends on how much rides on the decision. These rules of thumb are widely used, and they are the bands ExamQR's analysis reports apply to Cronbach's alpha:

A very high alpha, above about 0.95 on a short test, is not always good news: it can mean the questions are near-copies of each other and cover a narrow slice of the subject.

How to make a test more reliable

• Add good questions. Reliability rises with test length. By the Spearman-Brown formula, doubling a test with a reliability of 0.70 using similar questions raises it to about 0.82 (2 × 0.70 ÷ 1.70).
• Remove or rewrite questions that do not discriminate. A question that strong and weak candidates answer equally well adds noise, not information. Its discrimination, usually the point-biserial correlation between the question and the total score, should be positive; values above about 0.20 are generally acceptable and above 0.30 good. A negative value usually means a wrong answer key or a misleading question. More in item analysis: difficulty and discrimination.
• Fix distractors. A wrong option that nobody picks is not doing its job; one that attracts many strong candidates may be arguably correct.
• Mark consistently. For written answers, use a rubric and have a second marker check a sample.
• Standardise conditions. Same time limit, same instructions, no help available to some candidates and not others.

Validity: do the scores mean what you claim?

The current professional view, set out in the Standards for Educational and Psychological Testing, is that validity is not a property of a test. It belongs to a particular interpretation and use of its scores. The same English test can be valid for placing students into classes and not valid for deciding whether a nurse can work safely in an English-speaking hospital. So the right question is never "is this test valid?" but "is it valid to use these scores to make this decision about these people?"

That question is answered with evidence of several kinds:

• Content: do the questions cover the knowledge and skills the decision depends on, in sensible proportions? A written blueprint (topics × difficulty or skill level, with the number of questions in each cell) is the main tool here, and having subject experts review the questions against it is the main evidence.
• Response processes: do candidates answer by using the skill you intend? A maths question that most people get wrong because of its long, tricky wording is measuring reading, not maths.
• Internal structure: do the questions behave as the claimed structure says? If a test reports separate scores for listening and reading, the listening questions should relate more strongly to each other than to the reading ones.
• Relations to other measures: do scores agree with other evidence of the same skill, and predict what they are supposed to predict? A hiring test should relate to later job performance; a placement test, to how students then do in the class they were placed in.
• Consequences: does using the test lead to the intended outcomes without unfair side effects for particular groups?

Common threats to validity

• Testing the wrong thing: only recall questions in an exam meant to certify that people can apply a procedure.
• Irrelevant difficulty: heavy reading load, unfamiliar formats or a time limit so tight that speed decides the result when speed is not the skill.
• Clues to the answer: the correct option is always the longest, or grammar gives it away. Candidates who spot the pattern score higher without knowing more.
• Cheating and leaked questions: if answers circulate, scores measure access to the answers. Different papers per candidate, question banks large enough to vary, and supervision protect validity directly.
• Unreliable scores: every source of noise above is also a threat to validity.

How the two fit together

Reliability is necessary for validity but not sufficient. The link can be made precise: the correlation between test scores and anything else, such as later job performance, cannot be higher than the square root of the test's reliability. With a reliability of 0.64, scores can never correlate above 0.80 with the outcome they are meant to predict, however well the questions are chosen. Improving reliability raises the ceiling; only good design and evidence get you close to it.

In practice, work in this order: decide what the scores are for, write a blueprint that matches that purpose, write and review questions against it, give the test under consistent conditions, then check the statistics and fix what they reveal.

Check your own test

After each exam, look at four numbers before you publish results or reuse questions:

1. Cronbach's alpha or KR-20, against the table above for the stakes of your decision.
2. The SEM, and how many candidates sit within one SEM of the pass mark.
3. Discrimination for every question: rewrite or drop those below about 0.20, and check the answer key of any negative one first.
4. Distractor choices: options nobody chooses, and wrong options chosen by many strong candidates.

Frequently asked questions

Can a test be valid but not reliable?

Not really. If scores are mostly noise, they cannot support any decision well, so low reliability always limits validity. A test can, however, be reliable but not valid: consistent scores that measure the wrong thing.

Is Cronbach's alpha a measure of validity?

No. Alpha only says how consistently the questions measure something together. It does not say what that something is, or whether it is what you intended.

What is the difference between KR-20 and Cronbach's alpha?

KR-20 is the special case of alpha for questions scored as right or wrong. On such tests the two give the same value; alpha also works for questions with partial credit.

How many candidates do I need to calculate reliability?

The formula works with small groups, but the estimate is unstable below a few dozen candidates. Treat results from a small class as a rough signal, and look again when more candidates have taken the same questions.

Sources

• American Educational Research Association, American Psychological Association and National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. Washington, DC: AERA.
• Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334.
• Kuder, G. F. and Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2(3), 151–160.
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed.). New York: American Council on Education and Macmillan.