Item analysis: difficulty and discrimination explained
Item analysis is a set of statistics, calculated for each question after a test, that shows how the question performed. The three you need are difficulty (the proportion of candidates who answered correctly), discrimination (whether candidates who did well on the whole test were more likely to get the question right) and distractor analysis (who chose each wrong option).
Together they tell you which questions to keep, which to fix and which to drop before you use them again.
A question can look perfectly reasonable on paper and still fail in the exam room: the key is wrong, two options are both defensible, or a clue in the wording gives the answer away. You will rarely spot these problems by rereading the question. You will spot them in the answers candidates gave. This article explains the numbers, shows how to calculate them by hand on a small example, and gives the rules of thumb used to act on them.
Item difficulty: the p-value
The difficulty index, usually written p, is simply the proportion of candidates who answered the question correctly. If 22 of 40 candidates got it right, p = 22 ÷ 40 = 0.55. The name is counter-intuitive: a higher p means an easier question.
| p-value | Reading |
|---|---|
| Above 0.90 | Very easy |
| 0.71 to 0.90 | Easy |
| 0.30 to 0.70 | Medium |
| 0.10 to 0.29 | Hard |
| Below 0.10 | Very hard |
These are the bands ExamQR's analysis report uses. Which range you want depends on the purpose of the test:
- Tests that rank or select candidates, such as hiring tests and competitive exams, need questions that separate people. Questions of medium difficulty do this best, because a question almost everyone answers correctly, or almost no one does, cannot tell candidates apart. For four-option multiple-choice questions, where a candidate guessing at random gets 25% right, the classic recommendation is an average p of about 0.62, halfway between chance and 1.
- Mastery and certification tests check that candidates know what they must know. A p of 0.95 on an essential safety question is not a defect; it may be exactly what you want. Do not drop such a question for being easy.
Remember that p describes the question with this group. The same question is easier for a stronger class. A p-value from twenty candidates is a rough indication; from two hundred it is a dependable one.
Item discrimination: does the question separate strong from weak candidates?
A good question is answered correctly more often by candidates who know the subject well than by those who do not. Discrimination measures exactly that. There are two common ways to calculate it.
The discrimination index D (upper and lower groups)
This is the method you can do by hand:
- Rank candidates by total score.
- Take the top 27% as the upper group and the bottom 27% as the lower group. (27% is the classic choice: groups large enough to be stable, far enough apart to show a difference.)
- For each question, D = (correct in upper group − correct in lower group) ÷ size of one group.
With 40 candidates, each group has 11 people. If 9 of the upper group and 3 of the lower group answer a question correctly, D = (9 − 3) ÷ 11 = 0.55. D runs from −1 to +1.
The point-biserial correlation
Software usually reports the point-biserial correlation (rpb) instead: the correlation between getting this question right (1 or 0) and the total score, using every candidate rather than only the top and bottom groups. It is more precise, and it is what ExamQR reports as discrimination. Its values are generally a little lower than D for the same question, so do not compare the two scales directly.
| Discrimination (rpb) | Reading | What to do |
|---|---|---|
| 0.40 and above | Very good | Keep |
| 0.30 to 0.39 | Good | Keep |
| 0.20 to 0.29 | Moderate | Keep |
| 0.10 to 0.19 | Low | Consider improving the wording and distractors |
| 0 to 0.09 | Very low | Revise before reusing |
| Below 0 | Negative | Check the answer key first |
A negative value means candidates with low total scores got the question right more often than those with high ones. The most common cause is simple: the answer key is wrong. Other causes are a question with two correct answers, a trick in the wording that strong candidates overthink, or content that was not taught. Check these before anything else, because a wrong key also changes every candidate's score.
Difficulty limits discrimination. A question that 98% of candidates answer correctly cannot discriminate much, because there is almost no one left to separate. So read the two numbers together, not one at a time.
Distractor analysis: what the wrong options tell you
In a multiple-choice question, the wrong options, called distractors, have a job: to attract candidates who do not know the answer. Counting how often each option was chosen, split by upper and lower group, shows whether they do.
A good distractor:
- is chosen by a reasonable share of candidates. An option almost nobody picks is not working, and the question effectively has one option fewer, which makes guessing easier. ExamQR flags a distractor chosen by fewer than 5% of candidates once there are at least 30 of them;
- is chosen more by the lower group than by the upper group. In correlation terms, its point-biserial is negative. A distractor that attracts strong candidates (a positive correlation) usually means it is arguably correct, or the question is ambiguous. ExamQR flags distractors whose correlation with the total score is above 0.05.
A worked example
Forty candidates sat a test; the upper and lower groups have 11 candidates each. Here are two four-option questions. The key is in bold.
| Question 7 | A | B | C | D |
|---|---|---|---|---|
| Upper group (11) | 1 | 9 | 1 | 0 |
| Lower group (11) | 3 | 3 | 4 | 1 |
| All candidates (40) | 6 | 22 | 10 | 2 |
p = 22 ÷ 40 = 0.55, medium. D = (9 − 3) ÷ 11 = 0.55, strong. Distractors A and C attract mainly the lower group, as they should. Only option D is weak: just 2 of 40 candidates (5%) chose it, right at the usual threshold. Verdict: keep the question, and write a more plausible option D next time.
| Question 12 | A | B | C | D |
|---|---|---|---|---|
| Upper group (11) | 3 | 0 | 7 | 1 |
| Lower group (11) | 6 | 2 | 2 | 1 |
| All candidates (40) | 17 | 4 | 16 | 3 |
p = 17 ÷ 40 = 0.43, which looks fine on its own. But D = (3 − 6) ÷ 11 = −0.27: the lower group did better than the upper group. Seven of the eleven strongest candidates chose C. Either C is the correct answer and the key is wrong, or the question has two defensible answers. Verdict: check the key and the wording before scoring, then fix or drop the question.
Keep, fix or drop
ExamQR's report combines difficulty and discrimination into one recommendation per question, using these rules:
| Recommendation | When |
|---|---|
| Keep (excellent) | Discrimination 0.30 or more, and p between 0.20 and 0.80 |
| Keep (good) | Discrimination 0.20 or more, and p between 0.10 and 0.90 |
| Consider improving | Statistics acceptable but below the two rows above, often with a weak distractor |
| Revise | Discrimination below 0.10, or p above 0.95 or below 0.05 |
| Remove | Negative discrimination, or discrimination below 0.10 together with p above 0.95 or below 0.05 |
Treat these as a starting point for a human decision, not as a verdict. Before you remove a question, ask whether it covers something the test must cover. Removing every hard question on one topic makes the statistics look better and the test less valid. When a question is needed, rewrite it rather than drop it.
Common mistakes
- Trusting small samples. With fewer than about 30 candidates, a single person can move p by several points and swing discrimination widely. Use small-class statistics to find suspects, not to convict them.
- Dropping easy questions from a mastery test. Low discrimination on a question everyone knows is expected and harmless.
- Ignoring unanswered questions at the end. If many candidates did not reach the last questions, those questions look hard because of the time limit, not their content.
- Reading discrimination on a test that measures several different things. Discrimination compares each question with the total score. If the test mixes, say, grammar and listening, a good listening question can look weak against a total dominated by grammar. Analyse each part separately when you can.
- Forgetting the scores. When you find a wrong key, correct it and rescore the exam, not only the question bank.
Beyond classical item analysis
Everything above belongs to classical test theory, where every statistic depends on the group that sat the test. Item response theory (IRT) estimates a difficulty (b), a discrimination (a) and, in the three-parameter model, a guessing level (c) for each question on a common scale that depends less on the particular group. It needs more candidates, usually several hundred for stable three-parameter estimates, and it is most useful for large question banks and adaptive tests. For most classroom, training and hiring tests, the classical statistics in this article are enough to find the questions that need attention. The relationship between these question statistics and the reliability of the whole test is explained in reliability vs validity.
Run an item analysis on your own test
ExamQR's free test analysis tool runs everything in this article on any multiple-choice test, including tests you did not run on ExamQR. Paste the answer key and the candidates' answers, one line per candidate (5 to 200 questions, 10 to 5,000 candidates). You get the difficulty and discrimination of every question with the recommendation above, the share and correlation of every option with weak distractors flagged, Cronbach's alpha for the whole test and an IRT model. Results stay in your browser session and are deleted after two hours. For exams run on ExamQR, each subject's analysis page shows the difficulty and discrimination of every question. More about test analysis.
Frequently asked questions
What is a good difficulty index?
For tests that rank candidates, most questions between 0.30 and 0.80, with an average around 0.6 for four-option questions. For mastery tests, easy questions on essential content are fine.
What is a good discrimination index?
A point-biserial of 0.20 or more is acceptable and 0.30 or more is good. With the upper-lower D index, a common guide is 0.40 or more very good, 0.30 to 0.39 good, 0.20 to 0.29 marginal and below 0.20 poor.
Can discrimination be negative?
Yes. It means weaker candidates answered correctly more often than stronger ones. Check the answer key first; then look for two correct options or misleading wording.
How many candidates do I need?
The calculation works from ten candidates, but treat the results as indicative below about 30 and as dependable from about 100. IRT needs more.
Sources
- Ebel, R. L. and Frisbie, D. A. (1991). Essentials of Educational Measurement (5th ed.). Englewood Cliffs, NJ: Prentice Hall.
- Haladyna, T. M., Downing, S. M. and Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333.
- Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17–24.
- Lord, F. M. (1952). The relation of the reliability of multiple-choice tests to the distribution of item difficulties. Psychometrika, 17(2), 181–194.