Duolingo English Test:

Security, Proctoring, and Security, Proctoring, and Accommodations

June 24, 2020 (8 pages)
Accommodations englishtest.duolingo.com/resources

Duolingo English Test

Introduction

The Duolingo English Test is a high stakes assessment of general English language ability.
It is a computer adaptive test (CAT) that is delivered online and on­demand. It takes
approximately one hour to complete from onboarding to test upload. This document
outlines the security and proctoring features and processes of the test during and after its
administration.

Onboarding

The onboarding process begins with a browser and plugin check. The test checks that the
test taker is accessing the test via one of the supported browsers: Chrome, Opera, QQ,
360, or another Chromium­based browser that satisfies minimum technical requirements.
Plugins are automatically detected to alert users to turn them off. For example, if a test
taker uses a browser­based spelling and grammar assistant on a daily basis and forgets to
turn it off, the test will remind them to do so. The test taker cannot start the test without
disabling plugins. The onboarding process continues with an automated check that the
required devices (i.e., microphone and webcam) are present, that the test has permission
to access them, and that they are usable. Next, the test taker confirms that they are not
wearing any headphones and demonstrates that their webcam, microphone, and speaker
are working.

Corresponding author:
Duolingo English Test
Duolingo, Inc. 5900 Penn Ave, Pittsburgh, PA 15206, USA
Email: [email protected]

© 2020 Duolingo, Inc. All rights Reserved.

2 Security, Proctoring, and Accommodations

Next, the test taker fills in their ID and demographic information. The test then
captures photos of the test taker and their ID for identity verification. Accepted forms
of identification include passports, driver’s licenses, and national IDs that fulfill the
following ID quality standards* :

1. ID must be government issued and have a photo, name, birthday, date of expiration,
date of issuance of the bearer, and issuing body
2. ID must be free of visible tampering or damage
3. ID must show the required security features for that type of ID. Examples include:
holograms, raised seals, barcode, or visible chip
4. ID must be an original document. Photocopies or any other form of reproduction
will not be accepted
5. ID must be valid on date on test administration. Expired IDs will not be accepted.
6. ID must have current, clear photo of the test taker. IDs with an unrecognizable
photo will not be accepted.
7. Photo must be printed directly on the ID card and may not be embedded or attached
to the ID via lamination or adhesive.
8. Name on the ID must match the test taker’s name. Use of shortened names or
nicknames may cause a delay in test results. Scores cannot be sent to institutions
under any other name than the one listed on the ID.

After ID collection, the test taker is shown the test rules in their chosen user interface (UI)
language. The rules are as follows:

1. Ensure you are always fully visible.

2. Do not leave the camera preview.
3. Do not look away from the screen.
4. Keep your ears visible and uncovered.
5. Do not leave your web browser. If your cursor leaves the window or you exit full­
screen mode, your test will not be certified.
6. Do not speak unless instructed.
7. Do not communicate with another person at any point.
8. Do not allow others in the room with you.
9. Do not use any outside reference material.
10. Do not use a phone or other device.
11. Do not write or read notes.
12. Always keep your microphone and camera enabled.

∗ Inaddition to the above mentioned accepted forms of ID, UN Refugee documents are accepted, despite not
fulfilling all of the ID quality standards. These documents go through a special vetting process and can be
submitted by contacting Duolingo English Test support.

© 2020 Duolingo, Inc

 3

After reviewing the rules, they are asked to agree to the following:

• I have read and agree to the Terms of Service for the Duolingo English Test.
• I am the person whose ID and personal information has been provided for this test.
• The information I have provided is complete and accurate.
• I understand the rules for this test and I agree to follow them. I understand
that the proctoring supervisor will have the authority to make determinations in
their reasonable judgment as to whether I obeyed the rules and that Duolingo’s
determinations, subject to any appeals process Duolingo may decide to offer in
their sole discretion, are final.
• I understand that failure to follow the rules will prevent my results from being
certified. Depending on the circumstances I may or may not be allowed to retest
and in the case of certain violations I may be permanently banned from the testing
platform.
• I understand that I am prohibited from sharing or reposting any of the test materials
in any manner and that if I do so it is a violation of the terms of service and
may result in my results being decertified and me being banned from the testing
platform.

Test Administration

Prohibited Actions

The Duolingo English Test takes a “deterrence and detection” approach to security. To
deter cheating, the test disables keyboard actions, context menus, and browser features
(e.g. spell check) that could afford an unfair advantage and requests that browser plugins
be disabled. It also limits suspicious actions (e.g. moving the mouse off­screen, switching
tabs, reactivation and use of browser plugins), and automatically ends if these actions
occur too many times or for too long. If the test ends because of such actions, the test
taker has two more free attempts before they must purchase a new test credit, and they
may purchase no more than two test credits in a rolling 30­day window. If the test taker
runs out of attempts due to non­malicious reasons (e.g. technical errors), they may contact
customer support to obtain additional test credits. Our detection methods are covered in
the Post Test Administration section of this document.

Item Security

The Duolingo English Test achieves “passive” item security by way of its adaptivity and
a large item pool. Because the test is a CAT, test forms are assembled as the test is
administered using an algorithm that matches item difficulty with estimates of test taker
ability. This means that, given a large enough item pool, no two test forms are the same.
This pool contains tens of thousands of items. We used authentic data sources to train

© 2020 Duolingo, Inc

4 Security, Proctoring, and Accommodations

machine learning (ML) models, and used these models to generate items. Humans then
reviewed these items for fairness and bias (Settles, LaFlair, and Hagiwara 2020).
The item pool is regularly refreshed. The Duolingo English Test team also constantly
monitors item exposure and test overlap rates. An item’s exposure is the portion of tests
on which the item appears. Test overlap is the average number of items that are shared
between any two test administrations (Chen, Ankenmann, and Spray 2003; Way 1998).
The current mean item exposure and mean test overlap rates for the Duolingo English
Test are 4% and 1% respectively† . To set thresholds for acceptable item exposure and test
overlap rates, the team considered two factors: stakes and access (Way 1998). Because
the Duolingo English Test is used for university admissions, its stakes are high. Since
it is administered remotely, the potential for test takers to illicitly obtain access to items
is also high. As a result, thresholds for item exposure (10%) and test overlap (15%) are
conservative (as suggested by Way 1998). When items pass the exposure threshold, they
are retired. Regular item bank refreshes keep the average test overlap rate well below the
overlap threshold. In addition, item response patterns are regularly monitored for changes
in score distribution that could be attributed to over­exposure.
Since the test is fully online, attackers may attempt to breach our item pool by gaining
access to our API/database or through coordinated test­taking. We prevent API/database
attacks by maintaining a secure API, sponsoring regular security reviews, and regulating
access to the item database. Communications between server and client (browser) are
encrypted, so that it is impossible for a third­party “sniffing attack”. Additionally, all
item grading is done on the server­side. Test takers never have access to correct answers
or item grades. We detect coordinated test­taking by monitoring for anomalous test­taking
patterns, scanning the internet for item repositories, and fielding whistleblower reports.
It is unlikely that item breaches that result in sharing of test items pose much of a risk to
the security of our test. Because of the large size of the item pool, it is very unlikely that
any test taker that sees breached items will receive those same items in a test. However,
if the breach affects test scores the items are retired from the paid test to the practice test.

Post Test Administration

Proctoring takes place within 48 hours of the test administration and makes up the bulk of
the review time. Since proctor time and attention is a finite resource, the test supplements
their expert judgement with automated tools to improve efficiency. Immediately after test
administration, an artificial intelligence (AI) algorithm conducts an automated round of
proctoring. Human proctors then consider the results of this AI proctoring to determine
whether rules have been broken.

† These rates are below the 11–14% range reported for other operational CATs like the GRE
(https://www.ets.org/gre)

© 2020 Duolingo, Inc

 5

AI Assisted-Proctoring

The first round of proctoring is conducted by AI algorithms. The purpose of these

algorithms is to evaluate characteristics of the test performance in order to identify
potential indicators of rule breaking and malicious behavior. This process leverages
computer vision and biometrics processes to evaluate test taker behavior, natural
language processing algorithms to examine responses to production questions, and
statistical procedures to analyze response patterns. The algorithms “raise flags” with an
accompanying indicator of their confidence in those flags, which are reviewed by human
proctors for final decision.

Expert Human Proctoring

The Duolingo English Test is proctored in a blind, anonymous, multi­round review

process. Each round of proctoring is performed by fluent English speakers. Later rounds
are performed by certified English as a Second Language (ESL) educators. At each round
of proctoring, tests are randomly assigned to proctors, and proctors lack the ability to
search for specific test takers or groups of test takers (e.g. test takers from a given region,
test completion time, or tests associated with a specific institution). Furthermore, proctors
are anonymous to one another, thus eliminating the possibility of proctors trusting certain
proctors from previous rounds more than others. If at any time different rounds of
proctoring do not agree, a special group of senior proctors makes the final decision.
Using the AI tools provided, proctors examine each section of the test for over 80 different
behaviors that are grouped in several different severity categories. The first category of
behaviors that proctors look for are ID verification issues. The name, birthday, expiration
date, ID type, and ID photo are examined, and proctors note any inconsistencies.
Following ID verification, proctors examine the test in its entirety for minor, major, or
malicious test rule infractions. An example of a minor infraction is the presence of
background noise or a device alert noise, which can be unintentional in nature. These
minor infractions are reviewed in subsequent rounds to determine if they where used to
gain an unfair advantage. Examples of a major infractions include the test taker wearing
headphones or looking away for too long. Major infractions result in a “rules broken”
certification decision. The most severe category of infractions, malicious infractions,
which include test takers receiving outside help, result in a “malicious behavior” ruling
and a ban from the Duolingo English Test.

Reasons a Test My Not Be Certified

At the end of the proctoring process, a test is either certified or not certified. In the former
case, the test taker is sent their scores, and their scores are sent to any institutions that the
test taker selects. In latter case, the test is rejected for one of the following reasons:

© 2020 Duolingo, Inc

6 Security, Proctoring, and Accommodations

• Technical error

– There was a malfunction with the webcam, microphone, or speakers.

– There was an uploading issue that caused a severe video or audio delay.

• ID problems

– The ID was cropped, obscured, blurry, illegible, missing entirely.

– The ID was expired or not an ID that we accept or it was a photocopy.
– The ID was not an ID we accept for a specific country.

• Rules broken

– The user wore headphones

– The user’s ears were covered.
– The room is too dark.
– The room is not private.
– The room is not quiet.
– The user looks away from the screen repeatedly.
– There is someone else speaking with the user.
– The user does not speak enough during the open­ended speaking section (must
speak for at least 30 seconds as instructed)

• Malicious behavior

– The user taking the test is not the person in the ID picture.
– The user has already completed two valid tests in the past 30 days on another
account.
– The user is lip syncing.
– The user is using an external device or tool.
– The user is writing on paper / using a pen or pencil.
– The user is receiving assistance from another person.
– The user took a screenshot of the test screen.

Retake Policy by Certification Decision

The general policy about Duolingo English Test purchases is that one test credit should
return one test result. Test takers have three attempts to use their credit toward a result.
These three attempts can be used by the test taker when their test does not return a
result due to technical errors or breaking rules (non­maliciously). If the issues are not
resolved over three attempts, test takers can contact support and ask that a fourth attempt
be credited to their account. If there are problems with the the ID that is submitted (and no
other issues with the test administration), the test taker can resubmit their ID and receive
a score without retaking the test. If malicious behavior is the reason for non­certification
of results, the test taker is blocked and is not allowed to retake the test.

© 2020 Duolingo, Inc

 7

Data Retention

In order to prevent cheating and rule­breaking on the test, personally identifiable

information (PII) is collected from test takers. This includes:

• Photos of passports and/or government IDs

• Legal names
• Date of birth
• IP address and location data
• Videos of test takers’ screens and faces as they take the test

The Duolingo English Test is fully compliant with the European Union’s General Data
Protection Regulation (GDPR). At a test taker’s request, we will either share with them
or delete any data we have collected, except in cases where sharing/deleting data would
compromise the security of the test (e.g. videos of test takers’ screens, as this would leak
items).
Of the PII that we collect, ID photos are the most sensitive. All ID photos are retained for
only four days in encrypted storage after they test administration. After those four days,
they are moved to an even more secure location, where only a select group of employees
are permitted to access them.
Credit card information is collected as part of the payment process, but none of it is
retained. It is forwarded directly to credit card processors, who are also certified GDPR­
compliant.

Accommodations

Currently, we offer the following accessibility accommodations upon request: allowance

for headphones, screen magnifiers, hearing aids/devices, and any type of ergonomic
mouse or keyboard required to complete the test. There is ongoing research into adapting
the test in order to accommodate other special needs. The test is currently undergoing
updates available in the summer and fall of 2020. These updates will add the following
accommodations: additional time, dyslexic­friendly fonts, and a magnified screen option.

© 2020 Duolingo, Inc

8 Security, Proctoring, and Accommodations

References

Chen, S, R D Ankenmann, and J A Spray. 2003. “Exploring the Relationship Between

Item Exposure Rate and Item Overlap Rate in Computerized Adaptive Testing.” Journal
of Educational Measurement 40: 129–45.
Settles, Burr, Geoffrey T LaFlair, and Masato Hagiwara. 2020. “Machine Learning–
Driven Language Assessment.” Transactions of the Association for Computational
Linguistics 8: 247–63. https://doi.org/10.1162/tacl/_a/_00310.
Way, W. D. 1998. “Protecting the Integrity of Computerized Testing Item Pools.”
Educational Measurement: Issues and Practice 17 (4): 17–27.

© 2020 Duolingo, Inc