Examination Report

Part 1 Fellowship of the Royal College of

Ophthalmologists (FRCOphth)
Examination
January 2020
Matthew Turner, David Budzynski, Ben Smith

Page 1 of 16 Commercial-in-Confidence 21 February 2020

Contents
1 Summary...............................................................................................................................................3
2 Multiple choice question (MCQ) paper..............................................................................................4
2.1 Paper statistics.............................................................................................................................5
2.2 Quality of questions.....................................................................................................................6
2.3 Standard setting...........................................................................................................................6
3 Constructed response question (CRQ) paper.....................................................................................8
3.1 Paper statistics.............................................................................................................................8
3.2 Standard setting..........................................................................................................................10
4 Overall Results....................................................................................................................................12
4.1 Comparison with previous Part 1 examinations.........................................................................13
4.2 Breakdown of results.................................................................................................................14
Appendix 1: Overall results for each deanery.......................................................................................16

Page 2 of 16 Commercial-in-Confidence 21 February 2020

1 Summary
The Part 1 Fellowship of the Royal College of Ophthalmologists (FRCOphth) examination took place in
January 2020. A total of 116 candidates sat the examination, of which 53 (46 per cent) fulfilled the criteria
required to pass the examination overall.

The pass rate for candidates in Ophthalmic Specialist Training (OST) is 47 per cent compared with a 49 per
cent pass rate for non-trainees.

The multiple choice question (MCQ) exam had a reliability of 0.97 and the constructed response question
(CRQ) exam had a reliability of 0.91. The correlation between the two examinations was 0.75.

Page 3 of 16 Commercial-in-Confidence 21 February 2020

2 Multiple choice question (MCQ) paper
The table below gives the paper contents compared with previous years.

Table 1: MCQ paper content

Miscellaneous
Anatomy/embryolog Optic Patholog Pharmacolog Physiolog
Date & Total
y s y y & genetics y
investigations
Oct
24 24 23 18 23 8 120
2014
Jan
24 24 23 18 23 8 120
2015
May
24 24 23 18 23 8 120
2015
Oct
24 24 23 18 23 8 120
2015
Jan
24 23 23 18 23 8 119*
2016
May
24 24 22 18 23 8 119*
2016
Oct
24 24 23 18 23 8 120
2016
Jan
24 24 22 18 23 8 119*
2017
May
24 24 23 18 23 7 119*
2017
May
24 24 23 18 23 8 120
2018
Oct
24 24 23 18 23 8 120
2018
Jan
22 24 23 18 22 8 117*
2019
Apr
24 24 22 18 23 8 119*
2019
Oct
24 23 23 18 23 8 119*
2019
Jan
24 24 23 18 23 8 120
2020
* = questions removed

Page 4 of 16 Commercial-in-Confidence 21 February 2020

2.1 Paper statistics
Table 2: MCQ paper summary statistics

Statistic Value Percentage

Mean score 60/120 50.0%
Median score 66/120 55.0%
Standard deviation 24.7 20.6%
Candidates 130
Reliability: Cronbach's alpha 0.97
Standard error of measurement (SEM) 4.54 3.8%
Range of marks 0 – 98 0.0% – 81.7%
Pass mark derived from standard setting 73/120 60.8%
Pass - 1 SEM 69/120 57.5%
Pass rate 45/130 34.6%

Figure 1: Distribution of marks – MCQ

The vertical line denotes the point on the mark distribution where the pass mark lies.

Page 5 of 16 Commercial-in-Confidence 21 February 2020

2.2 Quality of questions
The Speedwell data allows us to identify easy, moderate and difficult questions, and those which are good, poor
or perverse (negative) discriminators. Ideally, all questions should be moderately difficult and good
discriminators.

Table 3: MCQ paper quality

Discrimination
Negative Poor Good
<0 0-0.249 ≥0.250 Tota
%
Numbe Numbe Numbe l
% % %
r r r
Difficult <25% 1 0.8 11 9.2 3 2.5 15 12.5
25–
Facilit Moderate 0 0.0 9 7.5 87 72.5 96 80.0
75%
y
Easy ≥75% 0 0.0 0 0.0 9 7.5 9 7.5
Total 1 0.8 20 16.7 99 82.5 120 100.0

2.3 Standard setting

The pass mark for the paper was agreed using the Ebel method.

Table 4: MCQ Ebel categories

Difficu Eas
Moderate Total
lt y
Essential 0 15 54 69
Important 3 18 17 38
Supplementar
2 4 7 13
y
Total 5 37 78 120

The Part 1 FRCOphth subcommittee considered the success of a minimally competent candidate in each
category as below:

Table 5: MCQ Ebel categories – expert decision

Difficu Eas
Moderate
lt y
Essential 0.55 0.65 0.75
Important 0.45 0.50 0.55
Supplementar
0.25 0.25 0.25
y

Table 6: MCQ Ebel categories – expert decision

Difficu
Moderate Easy Total
lt
Essential 0.00 10.00 40.00 50.00
Important 1.00 9.00 9.00 20.00
Supplementar
0.00 1.00 2.00 3.00
y
Total 2.00 20.00 52.00 73.00
Page 6 of 16 Commercial-in-Confidence 21 February 2020
 Difficu
Moderate Easy Total
lt

The MCQ pass mark was 73/120 (61%)

Table 7: Comparison of pass marks and rates for previous MCQ papers

33% discrimination Facility

Reliabilit Poor Difficul Easy Nu
te Mean SE Standar Pass Good
y Negative (0- t Moderate (>75%
score M d setting mark (>0.250)
(KR 20) 0.249) (<25%) ) que
71
88 68 0.85 4.90 Ebel (60% 3 59 58 6 90 24
)
71
07 69 0.90 4.90 Ebel (59% 3 55 62 6 91 23
)
71
23 70 0.90 4.90 Ebel (60% 6 34 79 3 90 26
)
72
94 71 0.88 4.80 Ebel (60% 5 49 66 9 88 23
)
71
01 64 0.80 NA Ebel (60%
)
75
36 69 0.80 4.80 Ebel (63% 6 63 50 8 89 22
)
72
19 70 0.83 4.73 Ebel (60% 15 61 44 9 70 41
)
72
14 70 0.86 4.84 Ebel (60% 7 68 45 4 87 29
)
70
96 65 0.82 4.68 Ebel (60% 15 63 39 15 70 32
)
73
19 72 0.89 4.70 Ebel (61% 10 46 63 9 79 31
)
72
86 68 0.84 4.67 Ebel (61% 10 65 44 13 68 38
)
73
30 60 0.97 4.54 Ebel (61% 1 20 99 15 96 9
)

3 Constructed response question (CRQ) paper

The table below gives the paper contents.

Page 7 of 16 Commercial-in-Confidence 21 February 2020

Table 8: CRQ paper content

Questio
Subsections
n
1 5
2 7
3 4
4 5
5 4
6 3
7 4
8 6
9 4
10 6
11 6
12 10

3.1 Paper statistics

Table 9: CRQ paper summary statistics

Statistic Value Percentage

Mean score 62/120 51.7%
Median score 64/120 53.3%
Standard deviation 14.9 12.4%
Candidates 116
Reliability: Cronbach's alpha 0.91
Standard error of measurement (SEM) 4.51* 3.8%
Range of marks 10 – 96 8.3% – 80.0%
Pass mark derived from standard setting 62/120 51.7%
Pass - 1 SEM 58/120 48.3%
Pass rate 69/116 59.5%

*Note that the CRQ paper is scored out of 240, with two examiners each marking out of 120. In order to put
the score back on the same scale as the MRQ paper and give each equal weight, the mark out of 240 is halved
and so is the SEM. As such this SEM value is technically [SEM out of 240]/2.

Page 8 of 16 Commercial-in-Confidence 21 February 2020

 Figure 2: Distribution of marks – CRQ

The vertical line denotes the point on the mark distribution where the pass mark lies.

Two examiners marked each question in the CRQ papers and the average mark from each was used to produce
the candidate mark. Each question has a maximum possible 10 marks. Candidate performance was variable
for each question, with mean, median, minimum and maximum scores (with standard deviations) set out in
Table 10 below.

Page 9 of 16 Commercial-in-Confidence 21 February 2020

 Table 10: Results for each question

Questio Mea Media Mi Ma BC

Subject SD
n n n n x M
1 Anatomy 5.20 5.00 0 10 2.72 5.50
2 Pathology 4.63 5.00 1 10 1.66 4.50
3 Pathology 4.18 4.00 0 8 1.76 4.00
4 Optics* 4.84 5.00 0 9 1.96 4.50
5 Optics* 5.19 5.00 0 10 2.35 5.50
6 Optics   4.97 5.00 0 10 2.70 5.00
7 Optics* 2.39 2.00 0 8 2.15 4.00
8 Investigations 7.97 9.00 0 10 1.92 6.00
9 Investigations 5.86 6.00 0 10 2.83 5.00
10 Investigations 7.20 7.00 1 10 1.58 6.50
11 Investigations 5.91 6.00 0 10 1.98 6.00
12 Statistics 5.66 6.00 1 9 2.03 5.25

Candidates performed badly in or were particularly ill prepared for question 7 (Optics*).

3.2 Standard setting

The borderline candidate method was used to identify the pass mark for the CRQ. The examiners who marked
the CRQ paper were asked to allocate a mark according to the marking scheme provided and, in addition, class
the candidate's performance as a pass, fail or borderline. The sum of each median borderline mark was used to
produce the pass mark.

Table 11: CRQ standard setting

Examiner A Examiner B
Question Topic Fail Border Pass Sum of Fail Border Pass Sum of
no. no. no. MBM no. no. no. MBM
1 Anatomy 48 37 31 6 41 44 31 5
2 Pathology 50 33 33 5 47 42 27 5
3 Pathology 58 24 34 4 56 34 26 4
4 Optics* 36 35 45 5 20 41 55 4
5 Optics* 43 23 50 5 30 55 31 4
6 Optics   51 37 28 5 45 41 30 5
7 Optics* 74 12 30 4 92 7 17 4
8 Investigations 16 86 14 6 12 89 15 6
9 Investigations 26 60 30 5 40 55 21 6
10 Investigations 9 80 27 6 12 46 58 7
11 Investigations 37 53 26 6 49 30 37 7
12 Statistics 36 57 23 5 37 33 46 6
Total 484 537 371 62 481 517 394 63

Table 12: Comparison with previous years

Page 10 of 16 Commercial-in-Confidence 21 February 2020

 Correlation
Reliabilit SE Pass Pass
Date Mean score Median score
y M mark rate
with MCQ
Oct 2014 50% 52% 0.94 4.3 57% 38% 0.76
Jan 2015 58% 62% 0.92 4.6 61% 56% 0.77
May 2015 51% 52% 0.93 4.6 54% 49% 0.75
Oct 2015 48% 50% 0.94 4.3 59% 28% 0.81
Jan 2016 48% 50% 0.94 3.0 54% 32% 0.80
May 2016 51% 54% 0.94 4.5 56% 41% 0.85
Oct 2016 50% 50% 0.93 4.0 59% 30% 0.83
Jan 2017 49% 51% 0.92 4.0 51% 50% Unknown
May 2017 57% 58% 0.92 5.0 53% 67% 0.76
May 2018 57% 59% 0.93 8.1 54% 71% 0.78
Oct 2018 58% 60% 0.93 4.8 55% 68% 0.75
Jan 2019 50% 52% 0.93 4.3 49% 62% 0.71
Apr 2019 44% 44% 0.94 4.6 51% 35% 0.83
Oct 2019 46% 49% 0.92 4.7 51% 41% 0.75
Jan 2020 51% 53% 0.91 4.5 52% 59% 0.75

Page 11 of 16 Commercial-in-Confidence 21 February 2020

4 Overall Results
To pass the Part 1 FRCOphth examination candidates are required to both

1. obtain a combined mark from both papers that equals or exceeds the combined pass marks obtained by
the standard setting exercise explained above, and
2. obtain a mark in both papers that equals or exceeds the pass mark minus one standard error of
measurement for each paper.

A candidate is therefore allowed to compensate a poor performance in one paper by a very good performance in
the other paper. They cannot compensate for an extremely poor performance in one paper whatever the
combined mark.

The minimum mark required in order to meet standard 1 above for this examination was 135/240 (56 per cent).
The minimum mark required in each paper (to meet standard 2 above) was 69/120 in the MCQ paper and
58/120 in the CRQ paper.

Fifty three (46 per cent) gained a total mark that met both standards 1 and 2 above. Thirteen candidates
achieved 135/240 or greater but failed to achieve 69/120 in the MCQ paper. In total, 53 out of 116 (46 per cent)
candidates passed the examination.

Figure 3: Distribution of marks – Combined

The vertical line denotes the point on the mark distribution where the pass mark lies.

Page 12 of 16 Commercial-in-Confidence 21 February 2020

4.1 Comparison with previous Part 1 examinations
Table 13: Comparison with previous years

Examinatio Candidate % MCQ pass mark CRQ pass mark

Number passing
n s passed % %
Oct 2006 33 3 9 58 62
Jan 2007 24 4 16 60 43
May 2007 32 5 15 50 64
Oct 2007 56 13 23 51 59
Jan 2008 73 27 37 56 55
May 2008 66 16 24 57 48
Oct 2008 88 45 51 58 51
Jan 2009 79 37 47 61 57
Jul 2009 49 33 67 63 58
Oct 2009 101 56 56 62 56
Jan 2010 50 20 40 63 58
May 2010 79 31 39 60 57
Oct 2010 89 34 38 61 54
Jan 2011 62 23 37 59 58
May 2011 95 47 49 54 57
Oct 2011 122 63 52 56 56
Jan 2012 66 20 33 57 54
May 2012 104 53 51 56 58
Oct 2012 150 84 56 56 54
Jan 2013 91 47 52 57 53
May 2013 102 54 53 58 58
Oct 2013 151 65 43 58 60
Jan 2014 77 23 30 57 57
May 2014 119 55 46 58 56
Oct 2014 232 102 44 58 57
Jan 2015 89 50 56 58 61
May 2015 114 62 54 57 54
Oct 2015 188 57 30 59 59
Jan 2016 107 36 34 59 54
May 2016 123 61 50 60 56
Oct 2016 194 70 36 60 59
Jan 2017 101 38 38 60 51
May 2017 136 62 46 63 53
May 2018 119 64 54 60 54
Oct 2018 214 122 57 60 55
Jan 2019 96 37 39 60 50
Apr 2019 119 45 38 61 51
Oct 2019 186 89 48 61 51
Jan 2020 116 53 46 61 52

Table 14: Comparison to previous years

Page 13 of 16 Commercial-in-Confidence 21 February 2020

 Candidate Number Pass rate
Sitting
s passing (%)
January 1031 415 40
May 1089 510 47
October 1771 817 46
Total 3891 1742 45

4.2 Breakdown of results

Table 15: Breakdown of results by training number (%)

Faile
Training Passed Percentage Total
d
In OST 24 21 46.7 45
Not in
30 29 49.2 59
OST
Unknown 9 3 25.0 12
Total 63 53 45.7 116

Table 16: Breakdown of results by deanery

Country Deanery Failed Passed Total

East Midlands 2 1 3
East of England 3 1 4
East of Scotland 2 1 3
KSS (Kent, Surrey &
0 1 1
Sussex)
London 3 2 5
Mersey 1 1 2
North of Scotland 0 2 2
UK
North Western 0 1 1
Northern 0 1 1
Northern Ireland 0 1 1
Oxford 0 1 1
Peninsula (South West) 0 1 1
Wales 2 1 3
West Midlands 1 1 2
Yorkshire 3 1 4
Overseas Europe and Overseas 1 1 2
Total 18 18 36

Table 17: Breakdown of results by stage of training

Page 14 of 16 Commercial-in-Confidence 21 February 2020

 Stage Failed Passed Percentage Total
FY2 3 1 75.0 4
MO
1 0 100.0 1
ST5
OST1 10 12 45.5 22
OST2 7 7 50.0 14
OST3 2 0 100.0 2
OST5 0 1 0.0 1
OST6 1 0 100.0 1
Total 24 21 53.3 45

Page 15 of 16 Commercial-in-Confidence 21 February 2020

Appendix 1: Overall results for each deanery
Result data by deanery has been available since October 2010. The summary results for each deanery are listed
below.

Table 18: Cumulative pass by deanery

Total candidates Total Pass rate

Country Deanery
passed candidates %
East Midlands 40 86 47
East of England 55 122 45
East of Scotland 14 17 82
KSS (Kent, Surrey &
47 76 62
Sussex)
London 149 268 56
Mersey 46 108 43
North of Scotland 20 40 50
North Western 39 63 62
Northern 42 76 55
UK
Northern Ireland 33 82 40
Oxford 25 38 66
Peninsula (South West) 32 71 45
Severn 18 32 56
South East of Scotland 25 39 64
Wales 51 106 48
Wessex 49 105 47
West Midlands 81 177 46
West of Scotland 57 112 51
Yorkshire 58 96 60
Eire 6 18 33
Overseas
Europe and Overseas 23 52 44
Total 910 1784 51

Page 16 of 16 Commercial-in-Confidence 21 February 2020