Calculating a Generalized Kappa Statistic for Use With Multiple Raters

Calculations based on equations presented in Fleiss (Statistical Methods for Rates and Proportions, 1981, pp. 229-232)
(Copyright © 2004, 2009 Jason King, Wallace Judd. All rights reserved. Available at http://www.ccitonline.org/jking/homepage/kappa1.xls)

Directions: Enter rating data in the Rater1, Rater2, Rater3, Rater4, and Rater5 worksheets, as needed.
Edit the shaded numbers below to indicate the number of raters, items and categories.

Enter # of raters (m): 5 =N_Raters 0.885 = Generalized Kappa for Rater 1, … Rater n data
Enter # of items (n): 100 =N_Items
# of categories (k): 6

The table below summarizes the data entered in the corresponding spreadsheets. Highlighting indicates areas of disagreement.

Items CAT1 CAT2 CAT3 CAT4 CAT5 CAT6

1 0 0 5 0 0 0
2 0 0 0 1 4 0
3 0 5 0 0 0 0
4 0 0 1 4 0 0
5 0 0 0 0 0 5
6 0 0 0 0 5 0
7 0 0 0 0 5 0
8 4 1 0 0 0 0
9 0 4 1 0 0 0
10 0 0 0 0 5 0
11 0 0 5 0 0 0
12 0 0 0 5 0 0
13 0 0 0 0 5 0
14 0 1 3 1 0 0
15 0 0 0 0 0 5
16 0 0 0 5 0 0
17 0 0 5 0 0 0
18 0 0 4 0 1 0
19 0 4 0 0 1 0
20 0 0 0 0 0 5
21 0 0 4 1 0 0
22 0 5 0 0 0 0
23 0 0 4 1 0 0
24 0 0 0 0 0 5
25 0 0 0 5 0 0
26 0 0 5 0 0 0
27 0 0 0 0 5 0
28 0 0 5 0 0 0
29 0 1 4 0 0 0
30 0 5 0 0 0 0
31 0 1 4 0 0 0
32 0 0 0 5 0 0
33 0 0 0 5 0 0
34 0 0 0 5 0 0
35 0 0 5 0 0 0
36 5 0 0 0 0 0
37 0 0 0 0 0 5
38 1 0 0 4 0 0
39 0 0 0 0 0 5
40 4 0 0 1 0 0
41 0 5 0 0 0 0
42 0 0 0 0 5 0
43 0 0 5 0 0 0
44 0 0 0 0 5 0
45 0 0 0 5 0 0
46 0 0 0 5 0 0
47 5 0 0 0 0 0
48 0 0 0 0 0 5
49 0 0 0 0 1 4
50 0 5 0 0 0 0
51 0 0 0 0 0 5
52 0 5 0 0 0 0
53 0 1 4 0 0 0
54 0 0 0 0 0 5
55 0 0 0 5 0 0
56 0 0 0 5 0 0
57 0 0 5 0 0 0
58 0 5 0 0 0 0
59 0 5 0 0 0 0
60 0 0 4 1 0 0
61 0 5 0 0 0 0
62 4 0 0 1 0 0
63 0 0 5 0 0 0
64 5 0 0 0 0 0
65 0 0 0 0 0 5
66 0 0 0 0 0 5
67 5 0 0 0 0 0
68 0 0 0 0 5 0
69 0 5 0 0 0 0
70 0 0 0 5 0 0
71 0 0 0 0 5 0
72 1 0 4 0 0 0
73 0 4 0 0 0 1
74 0 0 0 0 0 5
75 0 0 0 0 0 5
76 0 0 0 0 5 0
77 5 0 0 0 0 0
78 5 0 0 0 0 0
79 5 0 0 0 0 0
80 0 0 5 0 0 0
81 0 0 0 0 0 5
82 0 0 0 5 0 0
83 0 5 0 0 0 0
84 0 1 0 0 4 0
85 0 0 0 0 5 0
86 0 5 0 0 0 0
87 0 0 0 0 5 0
88 0 0 0 5 0 0
89 0 4 0 0 1 0
90 0 5 0 0 0 0
91 0 5 0 0 0 0
 92 5 0 0 0 0 0
93 5 0 0 0 0 0
94 0 0 0 0 5 0
95 0 0 5 0 0 0
96 0 4 1 0 0 0
97 0 5 0 0 0 0
98 0 0 0 0 0 5
99 0 0 0 0 0 5
100 0 0 4 1 0 0
Sum_x2
59 101 97 81 77 85 2310 =Prod_Cats
Total # of ratings 20 44 54 40 24 8 500 =Sum_Cats
prop (p) 0.118 0.202 0.194 0.162 0.154 0.170 1.000 Row = Cat_Props
1 2 3 4 5 6
****************** 1.000 1.000 1.000 1.000 1.000 1.000
BY CATEGORY CAT1 CAT2 CAT3 CAT4 CAT5 CAT6
num = sum_xi*(m-xi) 20 44 54 40 24 8
den = nm(m-1)pq 208.2 322.4 312.7 271.5 260.6 282.2
gen kappa_cat1 = 0.904 0.864 0.827 0.853 0.908 0.972

******************
OVERALL
num = nm2 - sum_x2 190
den = nm(m-1)sum_pq 1657.6
gen kappa = 0.885
(p(e)) 0.171
(sum p3) 0.030

SEFleiss1a 0.015 SEFleiss2b 0.014

z= 60.723 z= 61.897
p calc = 0.000000 p calc = 0.000000
 CILower = 0.857 CILower = 0.857
CIUpper = 0.914 CIUpper = 0.913

a
This approximate standard error formula is based on Fleiss (Psychological Bulletin, 1971, Vol. 76, 378-382)
b
This approximate standard error formula is based on Fleiss, Nee & Landis (Psychological Bulletin, 1979, Vol. 86, 974-977)
Rater 1 Classifications

Item Cat
1 3
2 5
3 2
4 4
5 6
6 5
7 5
8 1
9 2
10 5
11 3
12 4
13 5
14 3
15 6
16 4
17 3
18 3
19 2
20 6
21 3
22 2
23 3
24 6
25 4
26 3
27 5
28 3
29 3
30 2
31 3
32 4
33 4
34 4
35 3
36 1
37 6
38 4
39 6
40 1
41 2
42 5
43 3
44 5
45 4
46 4
47 1
48 6
49 6
50 2
51 6
52 2
53 3
54 6
55 4
56 4
57 3
58 2
59 2
60 3
61 2
62 1
63 3
64 1
65 6
66 6
67 1
68 5
69 2
70 4
71 5
72 3
73 2
74 6
75 6
76 5
77 1
78 1
79 1
80 3
81 6
82 4
83 2
 84 5
85 5
86 2
87 5
88 4
89 2
90 2
91 2
92 1
93 1
94 5
95 3
96 2
97 2
98 6
99 6
100 3
Rater 2 Classifications

Item Cat
1 3
2 4
3 2
4 4
5 6
6 5
7 5
8 1
9 3
10 5
11 3
12 4
13 5
14 2
15 6
16 4
17 3
18 3
19 5
20 6
21 4
22 2
23 3
24 6
25 4
26 3
27 5
28 3
29 3
30 2
31 2
32 4
33 4
34 4
35 3
36 1
37 6
38 1
39 6
40 1
41 2
42 5
43 3
44 5
45 4
46 4
47 1
48 6
49 5
50 2
51 6
52 2
53 3
54 6
55 4
56 4
57 3
58 2
59 2
60 4
61 2
62 1
63 3
64 1
65 6
66 6
67 1
68 5
69 2
70 4
71 5
72 3
73 6
74 6
75 6
76 5
77 1
78 1
79 1
80 3
81 6
82 4
83 2
 84 2
85 5
86 2
87 5
88 4
89 2
90 2
91 2
92 1
93 1
94 5
95 3
96 2
97 2
98 6
99 6
100 4
Rater 3 Classifications

Item Cat
1 3
2 5
3 2
4 3
5 6
6 5
7 5
8 2
9 2
10 5
11 3
12 4
13 5
14 4
15 6
16 4
17 3
18 5
19 2
20 6
21 3
22 2
23 4
24 6
25 4
26 3
27 5
28 3
29 2
30 2
31 3
32 4
33 4
34 4
35 3
36 1
37 6
38 4
39 6
40 4
41 2
42 5
43 3
44 5
45 4
46 4
47 1
48 6
49 6
50 2
51 6
52 2
53 2
54 6
55 4
56 4
57 3
58 2
59 2
60 3
61 2
62 4
63 3
64 1
65 6
66 6
67 1
68 5
69 2
70 4
71 5
72 1
73 2
74 6
75 6
76 5
77 1
78 1
79 1
80 3
81 6
82 4
83 2
 84 5
85 5
86 2
87 5
88 4
89 5
90 2
91 2
92 1
93 1
94 5
95 3
96 3
97 2
98 6
99 6
100 3
Rater 4 Classifications

Item Cat
1 3
2 5
3 2
4 4
5 6
6 5
7 5
8 1
9 2
10 5
11 3
12 4
13 5
14 3
15 6
16 4
17 3
18 3
19 2
20 6
21 3
22 2
23 3
24 6
25 4
26 3
27 5
28 3
29 3
30 2
31 3
32 4
33 4
34 4
35 3
36 1
37 6
38 4
39 6
40 1
41 2
42 5
43 3
44 5
45 4
46 4
47 1
48 6
49 6
50 2
51 6
52 2
53 3
54 6
55 4
56 4
57 3
58 2
59 2
60 3
61 2
62 1
63 3
64 1
65 6
66 6
67 1
68 5
69 2
70 4
71 5
72 3
73 2
74 6
75 6
76 5
77 1
78 1
79 1
80 3
81 6
82 4
83 2
 84 5
85 5
86 2
87 5
88 4
89 2
90 2
91 2
92 1
93 1
94 5
95 3
96 2
97 2
98 6
99 6
100 3
Rater 5 Classifications

Item Cat
1 3
2 5
3 2
4 4
5 6
6 5
7 5
8 1
9 2
10 5
11 3
12 4
13 5
14 3
15 6
16 4
17 3
18 3
19 2
20 6
21 3
22 2
23 3
24 6
25 4
26 3
27 5
28 3
29 3
30 2
31 3
32 4
33 4
34 4
35 3
36 1
37 6
38 4
39 6
40 1
41 2
42 5
43 3
44 5
45 4
46 4
47 1
48 6
49 6
50 2
51 6
52 2
53 3
54 6
55 4
56 4
57 3
58 2
59 2
60 3
61 2
62 1
63 3
64 1
65 6
66 6
67 1
68 5
69 2
70 4
71 5
72 3
73 2
74 6
75 6
76 5
77 1
78 1
79 1
80 3
81 6
82 4
83 2
 84 5
85 5
86 2
87 5
88 4
89 2
90 2
91 2
92 1
93 1
94 5
95 3
96 2
97 2
98 6
99 6
100 3