Many people ask: Is the DET proctored? The short answer: Yes! But that doesn't actually tell the whole story.
It’s understandable that there’s some confusion around this topic. Most people are only familiar with the tests they’ve had to take in the past, and a lot has changed over the last few years!
TL;DR: Many high-stakes entrance exams like the SAT and the TOEFL that were once administered in test centers are pivoting to online delivery in the face of ongoing pandemic-related closures, while digital-first exams like the Duolingo English Test (DET) were designed to be on-demand from the start.
The shift to digital testing has many people wondering if these tests are secure against cheating—which is where proctors come into play. But if you’re not in-person, how does proctoring work? Very good question, but let’s start with…
What is proctoring?
Proctoring means watching someone taking a test, to ensure that they don’t cheat. Traditional or “legacy” style tests that are administered in test centers use “live proctoring,” meaning someone is responsible for supervising a group of test takers sitting an exam together in a room, over the course of several hours.
This does not translate well to digital-first exams like the DET. Because they’re delivered online, on-demand, 24 hours a day, all over the world, it wouldn’t be efficient to have someone ready to proctor every session in person, in real time. But, as it turns out, live proctoring actually has quite a few drawbacks. In fact, when we first began designing the Duolingo English Test, one of the first questions we asked ourselves was…
Is live proctoring the best way to supervise an exam?
Live proctoring relies on a chain of trust consisting of dozens of humans, where a single point of failure can disrupt the entire chain. From test distributors to administrators, proctors, and scorers, there are myriad ways that humans could act in bad faith to compromise test security.
Live test sessions are not recorded. Because there’s no way for a proctor to pay attention to all test takers at the same time, a given test taker is only being partially observed during the test administration process. There's no video or voice recording of the test being administered—which means there’s no evidence of test taker behavior other than the proctor’s narrative, based on their memory.
Because proctors and test takers are in the same room, they cannot be anonymous. This is risky for a few reasons. Because test administrators and proctors are often co-located in the same city, there’s a risk that parties could conspire to get around test security measures. There are several cases in mainstream media of lax test administration security due to paid off administrators and groups of organized test takers helping each other in the same location.
Tests are administered in groups, so there’s often no precise way to distinguish cheaters from honest test takers. This can lead to cheaters going unnoticed, and honest test takers being accused of nefarious behavior. A single instance of cheating can mean that an entire batch of test takers may be unable to get a valid score.
So, back to the DET: Our model does not include live proctoring. But it’s also true that that kind of proctoring cannot provide full oversight for tests that are taken in the digital realm—or in person.
Who is proctoring the Duolingo English Test?
Our team of proctors is made up of hundreds of people across the globe, and is split into three tiers, each providing a separate and important layer of security.
First tier proctors are highly skilled in ID verification and in detecting various test taker behaviors that could be potentially suspicious. In addition to being trained in similar security skills, our Second tier proctors are also applied ESL professionals who spend the majority of their proctoring time reviewing the interview portion of the test. Proctors in our third tier are senior team members with expert applied ESL knowledge as well as higher level security skills. They review our most complex test cases.
When hiring proctors, we look for people with expert attention to detail and behavioral assessment skills. Once they’re hired, they go through extensive training involving live process reviews, working 1:1 with an experienced senior proctor mentor, and receiving detailed feedback. Our tier two and three proctors are also evaluated to ensure their applied ESL expertise. All proctors, regardless of tier, go through several weeks of training before they are able to review tests independently.
How does Duolingo keep digital exams secure?
You can think of our approach as proctoring plus. As part of our digital-first approach to test development, we’ve combined human expertise with the power of AI for a comprehensive approach to exam supervision.
Instead of being observed by one person in real time, every single test session is recorded via the computer, microphone, keyboard and mouse, and then reviewed by multiple rounds of trained professionals in order to be validated. Only then does a test taker receive a certified score.
Proctors review test sessions asynchronously and remotely, and are anonymous to test takers and to one another throughout the review process, so there’s no way for test takers to influence or coerce them. Each test session is discrete, so one person’s attempt at cheating can’t invalidate another test taker’s score. Because each session is recorded, there’s more evidence available to determine whether or not cheating occurred—evidence that can be observed and reviewed by multiple, anonymous proctors.
Test administration doesn’t look like it used to, and nor does test security. Advances in assessment and machine learning technology have meant we can administer exams far more rapidly, and to far more people, via a digital platform. It only makes sense for test review to evolve along with it, and the Duolingo English Test will always lead the way in testing innovation!