See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/359919954

Analysis of CSAT English Reading Test

Preprint · October 2021

CITATIONS READS
0 424

1 author:

Mitchell Gillick
University of Illinois Springfield
6 PUBLICATIONS 1 CITATION

SEE PROFILE

All content following this page was uploaded by Mitchell Gillick on 13 April 2022.

The user has requested enhancement of the downloaded file.

Mitchell Gillick

University of Illinois, Springfield

EDL 584: Assessment of ESL Students

Dr. Hanfu Mi

October 11, 2021

The College Scholastic Ability Test, colloquially called the CSAT or Suneung, is

the Korean college entrance exam that is administered to all college applicants in South

Korea. It is known as one of the most difficult and controversial academic tests in the

world. The test is only administered one day a year, with five separate subjects tested

concurrently from 8:40 AM to 5:45 PM (KICE, 2021). The sections include math,

Korean, English, Chinese, history, and vocational or science sections. All high school

seniors take the test, which makes up a majority portion of their college admissions

portfolio. If students receive a low score or miss the test, they will have to delay college

for another year, putting them at a severe disadvantage in their career (Sharif, 2018).

All of the Korean education system is geared towards the CSAT. Students begin

learning English in 3rd grade or earlier to prepare for the English section. Parents enroll

their children in after-school academies and tutor programs in math, Korean, history,

Chinese, and sciences to prepare for the exam when they are in high school. A large

majority of students attend over 12 hours of school per day. Teaching hours had to be

reduced by law to end before 10 PM in 2016 in response to students attending classes

after midnight.

The CSAT has been met with strong criticism since its inception in 1993 (Kim &

Ma, 2012). The test conditions are high-pressure, which have been blamed as the

cause of numerous student suicides from both test preparation and less than

satisfactory results. Some attribute the decreased birth rates in Korea, lowest among

OECD countries, to the fact that academies and extracurriculars are so expensive that

parents cannot afford to have a child (Sharif, 2018). The recent creation of a “Susi” path
to universities which places a higher importance on student portfolios advantages the

wealthy who are able to send their children abroad, thus increasing their language skills

and resumes. Both the Suneung (CSAT based) and Susi (resume based) college

application paths favor the wealthy, who are able to afford academies and tutoring.

Therefore, the CSAT has been met with criticism that it is too difficult, and puts the

majority of the population at a disadvantage for college placement (Kim, Kim, & Loury,

2011).

This paper will assess the validity of the CSAT test as well as how it has evolved

over the years in response to criticism and research. Research has been compiled

every year of the CSAT by both the Korea Institute of Curriculum and Evaluation, the

organization that oversees the CSAT, and independent researchers, as well as various

news organizations around the world.

Analysis of Instrument

To analyze the CSAT English tests, guidelines set out by ETS will be reviewed

against the content of the test. ETS is a worldwide test provider noted for developing the

TOEFL, TOEIC, GRE, and Praxis tests, which are provided in over 180 countries. The

CSAT is developed and administered by the Korea Institute for Curriculum and

Evaluation (KICE). According to KICE, the “College Scholastic Ability Test (CSAT) is

designed to measure students’ achievement of National Curriculum standards and

scholastic ability required for college education.” (KICE, 2021). The subjects included

are Korean Language Arts, Mathematics, English, Korean History, Investigation, and

Second Foreign Languages/Chinese Characters & Classics. The English section is

choice answers. For the purpose of this paper, only the English reading portion will be

assessed. KICE claims the English portion is used “to measure the achievement of

National Curricular standards of English and the proficiency of English for college

education.”

The English reading portion consists of 45 multiple choice questions which are

divided among three general question types: understanding general ideas,

understanding details, and understanding logical relations (Lee, 2020). Within those

question types are items that test the gist and title, atmosphere and feeling, vocabulary

in context, correspondence and discrepancy, fill-in-the-blank, unrelated sentence,

sentence insertion, sequence, and summary writing. According to Lee, there are slight

fluctuations in the question types, most notably an increase in questions pertaining to

atmosphere and feeling, increasing from one question in 2015 to three questions in

2016, 2018, 2019, and 2020.

The test was changed from norm-referenced scoring to criterion-referenced

scoring in 2018. Using T-tests, one-way ANOVA, FRE, FKGL, Coh-Metrix L2 readability,

and Lexile measures, Lee showed that there have been statistically significant

increases in the difficulty of the test (Lee, 2020). Sentence length has increased, text

levels have increased from a 7th grade level to a 10th grade level, and the reading

passages in understanding logical relations are more difficult after the change to

criterion-referenced testing.

KICE develops and tests the CSAT independently and secretly. Reports claim

that around 500 educators from around the country are selected in secret and
sequestered for a month to create the test (Sharif, 2018). A large portion of the test

questions are selected from EBS (Educational Broadcasting System) books, up to 70%,

while other texts are pulled from various “real world and academic sources,” (KICE,

2021). There has actually been controversy in the selection of source material, with

some authors suing KICE for using their material without permission and in protest of

the difficult nature of the test. Anecdotally, it is said that one writer’s text was used for

the CSAT as a question on summary meaning, and the author disagreed with the

answer of the CSAT.

In general, the entire CSAT English test is made up without native English

speakers. The author of this paper has personally met with some of the heads of the

English departments at the top universities in Korea, and noted how their spoken

English was far from perfect, including their presentations and written correspondence.

English grammatical mistakes are common in professional print, including student

textbooks. Nearly all Korean-published texts reviewed contained some form of

mistakes, be it spelling, grammar, or vocabulary. This calls into serious question the

validity of the test, if those who are responsible for teaching, creating, and publishing

materials have English errors.

In fact, when the questions of previous tests are examined, they can be seen to

contain clunky, unnatural English patterns that mix British and American patterns

together. If the assessment is intended to test natural English abilities, not the

memorization of testing strategies, then it would reason that any native English speaker

above the 12th grade level would be able to answer the test with ease. But if one

compares the CSAT with standard SAT English questions, the difference in language is
clear. SAT questions are written by native English speakers using natural language, an

academic vocabulary, but a more natural structure. The CSAT text passages contain far

more instances of passive voice and clauses, adding unnecessary difficulty to the

language. The main issue many have with the CSAT is that it is unrealistically difficult

for EFL Korean high schoolers to be able to attain a level of literacy able to pass an

unrealistic test. It would be the same as SAT reading portions being written in Old

English. An immediate need is transparency on who makes the test and from what

sources the materials are chosen. It should be obvious that texts from native English

speakers should be preferred over Korean-made texts. In a student survey, respondents

answered that EBS and CSAT prep books helped them “somewhat” and “a lot,” but very

few students selected “very much,” (Kim & Ma, 2021.) In interviews, students that study

with the EBS textbooks do so because they know that the majority of questions come

from them, so they are more likely to pass the CSAT.

All of these issues lead to questioning the validity and reliability of the CSAT. In

2018 only 9 students out of 530,220 attained a perfect score on the CSAT overall, with

5.3% of test-takers placed on the first-tier in English, down from 15 perfects and 10.03%

first-tier students in 2017 (Lee, 2018). Given the enormous importance in public and

private education, as well as study time, it seems unlikely that only 5% of students will

be first-tier; not perfect, but missing less than 5 out of 45 questions on a multiple choice

test. This further points to the high difficulty of the test.

Analysis of Intended Population

applications for Koreans. English is a second language, taught in the national

curriculum from 3rd grade with increasing time allocation (Ministry of Education).

Students and teachers reported that the national curriculum is guided entirely by the

CSAT, although the Ministry of Education claims that the curriculum should focus on

“practical English, speaking, and culture of English speaking countries.” This leads to

the phenomenon of “teaching for the test,” where the majority of classroom instruction is

geared towards the CSAT, leaving little to no time spent on practical or enrichment

learning.

The divide between economic and geographic scores is also evident in test

scores. Admissions to universities based on CSAT scores are highly skewed towards

the wealthy and those living in Seoul (Kim, Kim, & Loury, 2011). In more expensive

areas, such as Gangnam, student admission to Seoul National University was 173 per

10,000 students, but in a poorer district in Seoul, Geumcheon-gu, admissions were only

18 per 10,000, which indicates a nearly ten-fold gap between the SES of applicants.

Kim and Ma (2012), noted the disparity between the reading levels of high school

English II textbooks and the reading texts on the CSAT. On average, the RGL reading

level of the 2012 CSAT was 10.14, corresponding to tenth-grade native reading level.

The high school textbooks that students learn from were ranked as 7.48, leaving a 2

and a half year gap between what is tested and what is taught. The only way for

students to bridge the gap is by attending private education academies, which is one of

the leading causes of differences between test scores and college acceptance.
Assessment alignment

It is fairly clear now that there is little, if any, assessment alignment to the

intended material and purpose of the CSAT. The Ministry of Education claims that the

test is intended for someone with a high level of English competence to be able to pass,

without focusing on grammatical and technical factors (MOE, 2021). However,

numerous reviews of the testing material, language, reading levels, author responses,

and independent statistical analyses has proven that a fluent competence in English is

not enough. Instead, the test is entirely based on testing strategies, where students

learn the question types, how to scan for keywords, eliminate the incorrect answers,

and answer based on what the test makers want, not what is necessarily correct or

obvious.

Although the exam is made entirely for Korean natives with English as a foreign

language, the test is not highly modified for English language learners. On average, one

or two words per passage are translated on the actual test, but there is a high

occurrence of jargon, technical language, and idioms or colloquialisms that would not be

easily understood by foreign language learners. One key issue might be the use of

native reading levels, instead of basing the text on what could be expected of a foreign

reading level or second language learner. There are few if any commonly used metrics

to analyze the reading level for an ELL, and this could aid in the creation of future CSAT

exams.

Current reading level scores generally incorporate the number of syllables or

letters per word and the length of sentences. This method is blind to regional idioms and

vocabulary. For example, children’s books are often graded the same level because of
the few numbers of words per page, and increase with the length of the words, usually

based on 6 letter words or longer. However, words such as however, because, dinosaur,

together, computer, glasses, and notebook are all 6 letters or longer, but arguably are

low-level vocabulary words. Words such as qualm, nix, per, chide, and Id are short

words with far greater complexity and nuance in meaning. A book composed of short,

high level words could easily be graded as a lower level than a book with elementary

vocabulary simply because of the length of the text. The same follows for difficulties in

understanding regional discourse, such as y’all, shoulda, done been, porch, or supper. It

is common to find that English language learners struggle with a text which should be

age appropriate because the structure and language used in the book is cultural.

Supper has different meanings depending on if you are in the South, California, Maine,

or England, and what century it was written in. These features and ambiguities of

English place unnecessary burden and difficulty on the CSAT, and texts should be

selected to reflect that all of the test takers (statistically 99%) learned English as a

foreign language.

Testing Environment

The test date for the CSAT is a national date that imposes restrictions across the

country to ensure testing conditions (Sharif, 2018). Flights are re-routed and canceled,

all military drills are paused, police escorts are available to students, subway hours are

extended, work hours are altered, and the use of car horns is illegal. All of these sound

measures are in place during the time period on the testing date (there is only one date

per year for the CSAT) for the sole purpose of the English listening portion of the test.
Student phones are confiscated and stored in a soundproof room, and proctors are

required to wear rubber-bottomed shoes so the sound of their walking is not distracting.

Since COVID-19, thousands of students have taken the CSAT in hospital beds and

quarantined rooms, since no exceptions are made for missing the testing date.

Students arrive at testing centers to cheering crowds of family and supporters

who give sweet snacks and encourage students. Upon entering, students pass through

metal detectors and forfeit any digital watch, phone, or sound making device. The test is

extremely high pressure. The schedule of the test is laid out on KICE’s website,

showing that students check in by 8:10 AM, then take each test in scheduled order.

There are 80 minutes for the Korean test, a 20 minute break, 100 minutes for math, a

50 minute lunch break from 12:10-1:00, 70 minutes for English, a 20 minute break, then

electives testing, an 18 minute break, and finally a Second foreign language test from

5:05 - 5:45 PM. The effects of back-to-back testing has been reported as detrimental to

students’ mental health as well as testing ability, which is shown by there only being 5

students out of half of a million who got perfect scores on all of their tests.

The high stakes test environment has also been linked to youth suicide, with

numerous deaths surrounding the test directly related to and blamed on the CSAT. This

is a shocking effect of rigid and outdated testing practices, which has been minimally

addressed by the Ministry of Education and KICE. The MOE states each year that there

will be changes to the exam and national curriculum to lower test pressures, but several

key causes of stress remain: the CSAT is all only on one day a year, less time each year

is devoted to the subjects in the CSAT, restrictions on time spent in academies and
study is limited, and the test gets harder each year. If the MOE wanted to relieve the

stress, all of those factors must be addressed directly.

Suggested improvements would be having each subject tested on a different day.

Students would be able to devote more time preparing for one subject. The material

covered should be based on national textbooks and curriculum exclusively, as testing

on anything outside of that would disadvantage low-income students. The test should

be offered twice a year, so if students are unable to take the exam due to unforeseen

circumstances, they do not have to delay university enrollment for an entire year.

Predictions of Experiences

Students report that the CSAT is more difficult than the preparatory textbooks

and classes (Kim & Ma, 2012). Due to the high-pressure environment and the difficulty,

the nation in general has negative experiences with the CSAT. This points to a serious

need for an overhaul of the exam to be in line with contemporary research, standards,

and practices. As of now, the only guaranteed way to get a high score on the English

portion of the CSAT is to have lived abroad, have parents with graduate degrees, live in

a wealthy area of Seoul, and attend private academies. All students know this, and it is

extremely demoralizing for test takers.

As mentioned by Hong, Bae, and Wu (2019), teachers need to teach for

high-stakes testing. The authors examined the mathematics portion of the exam, and

found that “[a]bout 44.4% to 49.7% of mathematical tasks in the CSAT are aligned to

what the Korean mathematics curriculum standards expect students to know.” This
means that the majority of questions on the test are not something that they have likely

learned from regular public education.

Conclusion

Given the overwhelming negative reactions to the CSAT from educators and

students, the misalignment between the test content and national curriculum, the high

pressure testing conditions, the disadvantages to lower-income and rural populations,

the misinterpretation and accuracy of the actual questions, the secretive creation of the

test without outside validity testing, and the increasing difficulty in the reading levels, it

can only be concluded that the CSAT English test is not a valid test for high school EFL

students as a measure of their English skills pertaining to college readiness. A severe

overhaul of the CSAT would be needed to meet the requirements of an actual test of

English skill.

Thousands of researchers, news organizations, and articles are published each

year regarding the way that all of Korean society and life is somehow affected by the

CSAT. It would be in the best national interest for both language learning and quality of

life, that the test be developed differently. Using materials from the national curriculum

exclusively, aligning the reading level with a realistic expectation of the English levels

used at Korean universities, spreading the testing across multiple days, and involving

native English speakers in the development process would all be desirable features of

future CSAT exams. A possible remedy would be to simply adopt another universally

accepted language exam, such as those available from ETS, such as the TOEFL.
 The CSAT does not test what it is intended to test, which is practical and

academic English at the university freshman level, but instead tests memorization of

patterns and rules that do not benefit students. In personal interviews, this author

discussed with teachers, adults, and students if they used any English skills learned in

preparation for the CSAT in university or after, and the overwhelming majority said no. A

closer evaluation of real-world university and high school materials based on English

needs should be examined and applied to the CSAT.

Works Cited

College Scholastic Ability Test. Korea Institute for Curriculum and Evaluation. (2021).

Retrieved October 9, 2021, from

https://www.kice.re.kr/sub/info.do?m=0205&s=english.

ETS. (2021). Guidelines for the assessment of English language learners.

https://www.ets.org/s/about/pdf/ell_guidelines.pdf

Hong, DS, Bae, Y., & Wu, Y.-F. (2019). Alignment between National College Entrance

Examinations and Mathematics Curriculum Standards: A Comparative Analysis.

Journal of the Korean Society for Mathematical Education Series D: Mathematics

Education Research , 22 (3), 153–174.

https://doi.org/10.7468/JKSMED.2019.22.3.153

Lee, M. (2018, December 5). Only 9 of half-million test takers get perfect 'suneung'

score. Korea Times. Retrieved October 10, 2021, from

https://www.koreatimes.co.kr/www/nation/2018/12/181_259894.html.
Lee, J. (2020). A Comparison of Text Difficulty in 2015-2020 CSAT English Reading

Passages between Testing Types and among Question Types. Journal of the

Korea English Education Society, Vol. 19, No. 2, 19(2), 21-43.

Kim, Y., & Ma, J. H. (2012). Text readability of the College Scholastic Ability Test and

high school English II textbooks. English Teaching, 67(4), 195–218.

https://doi.org/10.15858/engtea.67.4.201212.195

Kim, Y. C., Kim, Y.J., & Loury, G. (2011). Widening gap in college admission and

improving equal opportunity in South Korea. Brown University.

https://www.brown.edu/Departments/Economics/Faculty/Glenn_Loury/louryhome

page/cvandbio/Widening%20Gap%20in%20College%20Admission%20in%20So

uth%20Korea.pdf

Ministry of Education. (2021). Higher Education. Retrieved October 9, 2021,

fromhttp://english.moe.go.kr/sub/info.do?m=020105&s=english.

OECD. (November, 2016). Education policy outlook: Korea.

https://www.oecd.org/education/Education-Policy-Outlook-Korea.pdf

Sharif, H. (2018, November 26). Suneung: The day silence falls over South Korea. BBC

News. Retrieved October 9, 2021, from

https://www.bbc.com/news/world-asia-46181240.

Appendix
View publication stats