C. Item Analysis http://fcit.usf.edu/assessment/selected/responsec.

html

After you create your objective assessment items and give your test, how can you be sure that
the items are appropriate -- not too difficult and not too easy? How will you know if the test
effectively differentiates between students who do well on the overall test and those who do
not? An item analysis is a valuable, yet relatively easy, procedure that teachers can use to
answer both of these questions.

To determine the difficulty level of test items, a measure called the Difficulty Index is used.
This measure asks teachers to calculate the proportion of students who answered the test item
accurately. By looking at each alternative (for multiple choice), we can also find out if there
are answer choices that should be replaced. For example, let's say you gave a multiple choice
quiz and there were four answer choices (A, B, C, and D). The following table illustrates how
many students selected each answer choice for Question #1 and #2.

Question A B C D
#1 0 3 24* 3
#2 12* 13 3 2

* Denotes correct answer.

For Question #1, we can see that A was not a very good distractor -- no one selected that
answer. We can also compute the difficulty of the item by dividing the number of students
who choose the correct answer (24) by the number of total students (30). Using this formula,
the difficulty of Question #1 (referred to as p) is equal to 24/30 or .80. A rough "rule-of-
thumb" is that if the item difficulty is more than .75, it is an easy item; if the difficulty is
below .25, it is a difficult item. Given these parameters, this item could be regarded
moderately easy -- lots (80%) of students got it correct. In contrast, Question #2 is much
more difficult (12/30 = .40). In fact, on Question #2, more students selected an incorrect
answer (B) than selected the correct answer (A). This item should be carefully analyzed to
ensure that B is an appropriate distractor.

Another measure, the Discrimination Index, refers to how well an assessment differentiates
between high and low scorers. In other words, you should be able to expect that the high-
performing students would select the correct answer for each question more often than the
low-performing students. If this is true, then the assessment is said to have a positive
discrimination index (between 0 and 1) -- indicating that students who received a high total
score chose the correct answer for a specific item more often than the students who had a
lower overall score. If, however, you find that more of the low-performing students got a
specific item correct, then the item has a negative discrimination index (between -1 and 0).
Let's look at an example.

Table 2 displays the results of ten questions on a quiz. Note that the students are arranged
with the top overall scorers at the top of the table.

1
 Total Questions
Student
Score (%) 1 2 3
Asif 90 1 0 1
Sam 90 1 0 1
Jill 80 0 0 1
Charlie 80 1 0 1
Sonya 70 1 0 1
Ruben 60 1 0 0
Clay 60 1 0 1
Kelley 50 1 1 0
Justin 50 1 1 0
Tonya 40 0 1 0

"1" indicates the answer was correct; "0" indicates it was incorrect.

Follow these steps to determine the Difficulty Index and the Discrimination Index.

1. After the students are arranged with the highest overall scores at the top, count the
number of students in the upper and lower group who got each item correct. For
Question #1, there were 4 students in the top half who got it correct, and 4 students in
the bottom half.
2. Determine the Difficulty Index by dividing the number who got it correct by the total
number of students. For Question #1, this would be 8/10 or p=.80.
3. Determine the Discrimination Index by subtracting the number of students in the
lower group who got the item correct from the number of students in the upper group
who got the item correct. Then, divide by the number of students in each group (in
this case, there are five in each group). For Question #1, that means you would
subtract 4 from 4, and divide by 5, which results in a Discrimination Index of 0.
4. The answers for Questions 1-3 are provided in Table 2.

# Correct (Upper # Correct (Lower Difficulty Discrimination

Item
group) group) (p) (D)
Question
4 4 .80 0
1
Question
0 3 .30 -0.6
2
Question
5 1 .60 0.8
3

2
Now that we have the table filled in, what does it mean? We can see that Question #2 had a
difficulty index of .30 (meaning it was quite difficult), and it also had a negative
discrimination index of -0.6 (meaning that the low-performing students were more likely to
get this item correct). This question should be carefully analyzed, and probably deleted or
changed. Our "best" overall question is Question 3, which had a moderate difficulty level
(.60), and discriminated extremely well (0.8).

Another consideration for an item analysis is the cognitive level that is being assessed. For
example, you might categorize the questions based on Bloom's taxonomy (perhaps grouping
questions that address Level I and those that address Level II). In this manner, you would be
able to determine if the difficulty index and discrimination index of those groups of questions
are appropriate. For example, you might note that the majority of the questions that demand
higher levels of thinking skills are too difficult or do not discriminate well. You could then
concentrate on improving those questions and focus your instructional strategies on higher-
level skills.

Try This

Now it's your turn. Try the same procedures using the data in the Item Analysis Worksheet.
When you are finished, you can check your answers by looking at the Answer Sheet.

 Item Analysis Worksheet

 Item Analysis Answer Sheet

3
 Item Analysis Worksheet
Ten students have taken an objective assessment. The quiz contained 10 questions. In
the table below, the students’ scores have been listed from high to low (Joe, Dave, Sujie,
Darrell, and Eliza are in the upper half). There are five students in the upper half and
five students in the lower half. The number“1” indicates a correct answer on the
question; a “0” indicates an incorrect answer.

Student Total Questions

Score
(%) 1 2 3 4 5 6 7 8 9 10
Name
Joe 100 1 1 1 1 1 1 1 1 1 1

Dave 90 1 1 1 1 1 1 1 1 0 1

Sujie 80 1 1 0 1 1 1 1 1 0 0

Darrell 70 0 1 1 1 1 1 0 1 0 1

Eliza 70 1 1 1 0 1 1 1 0 0 1

Zoe 60 1 1 1 0 1 1 0 1 0 0

Grace 60 0 1 1 0 1 1 0 1 0 1

Hannah 50 0 1 1 1 0 0 1 0 1 0

Ricky 40 1 1 1 0 1 0 0 0 0 1

Anita 30 0 1 0 0 0 1 0 0 1 0

Calculate the Difficulty Index (p) and the Discrimination Index (D) for each question.

# Correct # Correct
Difficulty
Item (Upper (Lower Discrimination (D)
(p)
group) group)

Question 1

Question 2

4
 Question 3

Question 4

Question 5

Question 6

Question 7

Question 8

Question 9

Question 10

Answer the following questions:

1.Which question was the easiest?

2. Which question was the most difficult?
3. Which item has the poorest discrimination?
4. Which questions would you eliminate first (if any) – why?

5
Answers

Calculate the Difficulty Index (p) and the Discrimination Index (D).

# Correct # Correct Difficulty Discrimination

Item
(Upper group) (Lower group) (p) (D)

Question 1 4 2 0.6 0.4

Question 2 5 5 1.0 0

Question 3 4 4 0.8 0

Question 4 4 1 0.5 0.6

Question 5 5 2 0.8 0.6

Question 6 5 3 0.8 0.4

Question 7 4 1 0.5 0.6

Question 8 4 2 0.6 0.4

Question 9 1 3 0.3 -0.4

Question 10 4 2 0.6 0.4

Answer the following questions:

1.Which question was the easiest? Question #2 (p=1.0)

2. Which question was the most difficult? Question #9 (p=.03 – only 30% of
the students got it correct)
3. Which item has the poorest discrimination? Question #9 (D=-0.4 – it has
negative discrimination – the lower students were more likely to get it correct)
4. Which questions would you eliminate first (if any) – why? Question #9 –
because of the negative discrimination.