Dan Isbell is an Associate Professor in Second Language Studies at the University of Hawaiʻi at Mānoa whose research bridges language assessment and second language acquisition. He investigates the validity of language tests—both high-stakes and classroom-based—and how they can be used to support learning.
Goodhart’s Law asserts “when a measure becomes a target, it ceases to be a good measure”.
While this well-known adage originates in economics, it has been widely applied to many other areas of life, including language tests. But why?
High-stakes language tests can unlock doors—to education, employment, and migration opportunities. They also cost a bit of money, too. Naturally, if you plan to take a test, you’ll likely know what score you need (i.e., your target) and will want to prepare as best you can to reach it.
How people go about preparing for a test matters. If people just learn ‘tricks’ to get higher scores without actually getting better at the thing the test measures, like English, then the measure ceases to be a good measure! On the other hand, if a test motivates people to develop their English skills, that’s a good thing, and it makes Goodhart’s Law less of a concern.
From decades of research on language test preparation, we also know that the features of a test can influence how people prepare for it. As a newer test, and one with features that distinguish it from many other standardized tests of English (such as human-supported automated item generation, computer adaptive delivery, and at-home test taking), we don’t yet have a detailed understanding of how people go about preparing for the Duolingo English Test (DET).
In what follows, I will provide a bit more background on language test preparation and share some findings from a study on DET preparation my team and I conducted.
There’s more than one way to prepare for a test
Test preparation activities can be classified into three main types:
- Type 1: Activities that help you improve the knowledge, skills, and abilities the test measures. For an English language test, this would be doing things to generally improve your English ability, like studying vocabulary or listening to the news.
- Type 2: Activities that help you perform your best on the test, usually related to familiarization with the test (such as learning about time limits and expected formats for responses) and minimizing potential distractions like anxiety. Taking a practice test falls into this category.
- Type 3: Activities that (might) help you get a higher score on the test, but in a way that tries to get around the skills that the test measures. This can include things like memorizing detailed templates for writing tasks, studying strategies for guessing, and even ‘hacks’ and tips for how to cheat on an exam.
The types and intensity of test preparation people engage in is often linked to the uses and characteristics of the test and its stakes.
Tests influence the way we study English
Maybe you know someone who spent a lot of money and many late nights cramming anatomical vocabulary to take a medical exam, or perhaps you’ve heard about intensive, years-long preparation in public schools and private academies for university entrance exams in places like India and South Korea. Where the stakes are high, the test preparation can be intensive and often focused on Type 2 and Type 3 activities.
In the world of testing, this phenomenon is called washback: the impact a test can have on the learning process. Washback can be positive if it motivates people to learn something useful, or negative if it motivates people to spend a lot of time doing things that are not useful outside of the test. While it’s not the job of a test provider to teach you English (although some test providers have an app for that or offer courses), test providers do wish to minimize negative washback so that test takers and other stakeholders like teachers, parents, and administrators see the test as a complement to their learning efforts, rather than an obstacle or intrusion.
Connecting washback to test preparation types, it’s fine to see lots of Type 1 activity, and Type 2 activities in small doses. Type 3 activity clearly corresponds to negative washback, and is hopefully minimal, if not nonexistent.
So, how do people prepare for the DET?
It’s quite clear that people planning to take the DET commonly take the DET Practice Test at least once, but beyond that we don't have a very detailed understanding of how people go about preparing for the test.
With a large team of researchers affiliated with the University of Hawaiʻi at Mānoa, I’ve been interviewing DET test takers and poring over test preparation resources to better understand the range of activities people engage in to get ready for the DET.
Our research has included students currently studying at universities in the US and Canada, as well as popular English-language YouTube channels and online courses. We also decided to include a focus on the East Asia region, where exam preparation is extremely prevalent and where a large portion of DET test takers are located (see the DET team’s report on scores and demographics).
Thanks to our research team’s language skills, we’ve been able to interview test takers, test prep coaches, and materials developers—39 people in total—and analyze test preparation materials produced in English, (Mandarin) Chinese, Japanese, and Korean, reviewing over 160 sources in total!
The verdict on washback
Our findings suggest that getting familiar with the test by taking the practice test, practicing specific test tasks, and reading/watching information about the tests (in other words, Type 2 preparation) is most commonly featured in preparation materials and mentioned by interviewees.
Doing things that develop general English skills (Type 1) were less commonly reported by interviewees or featured in test preparation materials. Focusing on tricks to get a better score (Type 3) was least common.
In a bit more detail, we also found some interesting links between the design and administration of the DET, and how people prepare for it:
- Studying Vocabulary (Type 1): Vocabulary resources are easy to find and learning vocabulary is a key part of language learning. For the DET, the presence of Yes/No Vocabulary and C-test tasks provided extra motivation to study vocabulary, above and beyond its usefulness in reading, writing, listening, and speaking.
- Typing practice (Type 2): With tasks like Dictation and extensive writing tasks that feature short time limits, developing speed in typing English is seen as important.
- Familiarization with Online Testing (Type 2): Resources that gave advice on how to set up a suitable testing environment and avoid suspicions of cheating, like keeping your gaze fixed on your monitor, were somewhat common.
- Gaming the Scoring System (Type 3): As a test entirely scored by computer, some preparation resources provide tips that are supposed to trick the scoring system into giving a higher score. Unfortunately, a lot of this advice (a) doesn’t help your real-world communication and (b) probably won’t make a difference on your DET score. For example, some of the less helpful advice we saw was to try to fill every second possible, no matter what, on speaking tasks, or to do everything you could to maximize the number of words you produce on writing tasks.
In the materials we analyzed and interviews we conducted, we saw evidence for both positive and negative washback through Type 1 and Type 3 preparation, respectively. We also saw lots of Type 2 preparation focused on test familiarization, which is potentially neutral. But it’s hard to generalize the washback effect of a test to every test taker.
Some people we interviewed tried the official practice test, watched a video or two about how to take the test, and then took their official DET in the same afternoon. This might be the ideal scenario for test preparation, even though there was no activity leading to increased English skills! On the other hand, some people prepared intensively for several weeks or longer, mixing Type 2 and Type 3 preparation.
Keeping test preparation positive
So what should you do? While everyone’s test taking journey will look a little different, I recommend aiming for a positive preparation experience: get familiar with the test, and focus on developing genuine English skills as needed to achieve your goals.
First, a large body of research on standardized tests suggests that taking a practice test is the single best thing you can do to ensure you perform your best on an official test, so take advantage of the official DET Practice Test, and watch a few videos on the DET YouTube channel to make sure you understand how to set up your room and computer for a secure official test.
If your typing speed in English using a keyboard is on the slower side, doing some typing practice wouldn’t be a bad idea (plus, this is something likely to be useful in real life!). If your target score is a bit above your current abilities, you’ll want to make a longer-term study plan that includes things like vocabulary and grammar study, but I also recommend practicing using English—something that was unfortunately mentioned a bit less commonly than studying vocabulary and grammar in our data.
Preparing for the DET can include fun activities like watching movies, reading the news, participating in a language exchange, and posting on social media. All of these will help grow your language skills!