AI plays an important role in how the Duolingo English Test (DET) measures English proficiency. But how does automated scoring actually work? Does the DET use ChatGPT to assign scores? Can AI evaluate different accents fairly? And do longer answers automatically earn higher scores?
As co-lead of the DET Test Scoring team, I hear questions like these from test takers, educators, and admissions professionals all the time. There's a lot of confusion around AI scoring, so I want to explain what our scoring models actually do, how human experts guide their development, and how we evaluate their accuracy and fairness.
Does the Duolingo English Test use ChatGPT to score responses?
No. We use specialized machine-learning models to evaluate speaking and writing responses.
Our models combine approximately 100 different features that measure specific aspects of English proficiency. Our assessment experts select these features based on what they tell us about a test taker's ability to communicate in English.
Some features use large language models (LLMs). For example, an LLM-based feature helps evaluate how coherently someone connects ideas in a response. We fine-tune and validate these features against expert human ratings of the specific quality they're designed to measure.
Each feature contributes evidence to the scoring process. A general-purpose chatbot or an LLM never makes an overall judgment about the quality of a test taker’s response.
How do human experts oversee automated scoring?
DET scoring is automated, but it isn't autonomous. Human experts design, train, validate, and monitor our scoring system. How does it work?
First Assessment scientists define what to measure. We identify the aspects of speaking and writing that reflect English proficiency and select features that capture those abilities. Then, human raters evaluate responses. A separate group of trained experts rates real test taker responses. These ratings provide the reference data we use to train and evaluate our scoring models. We test accuracy and fairness. Every model must pass rigorous checks before we approve it for use. Finally, we monitor performance. Once a model is in operation, we continuously evaluate its performance and investigate potential issues.
Human oversight also extends to test security. Our trained human proctors review every test session before scores are released to confirm that test rules were followed. These proctors evaluate the integrity of the testing session. They do not re-score speaking or writing responses.
This human-in-the-loop approach combines the consistency and scalability of automated scoring with expert judgment throughout the scoring process.
How does the DET make sure AI scoring is fair?
Fairness is a central consideration when we develop automated scoring systems. Our test takers come from different linguistic, geographic, and demographic backgrounds, and our scoring models need to evaluate their English proficiency consistently.
We address fairness throughout model development, beginning with the data we use to train the system.
Our training datasets include approximately 100,000 real test sessions each for speaking and writing. These datasets reflect a broad range of test takers: More than 100 first languages are represented, and the most common first language accounts for less than 17% of sessions. Throughout, gender representation is close to evenly balanced.
We also recruit expert raters from diverse national, geographic, and linguistic backgrounds, including people familiar with different varieties of English.
Before introducing a new model or scoring feature, we examine whether test takers with comparable proficiency receive systematically different scores based on characteristics such as first language, gender, or region.
When these analyses identify concerns, we can revise or remove a feature before it becomes part of the scoring system.
These fairness checks continue throughout the model's development and use.
Do longer speaking and writing responses get higher DET scores?
Longer responses don't automatically receive higher scores. We evaluate many aspects of language proficiency, including the quality and development of ideas.
For writing, our scoring features examine characteristics such as content and how effectively ideas are developed; coherence and connections between sentences; vocabulary use; and grammar, including more than 50 types of grammatical errors. For speaking, our models also examine features such as fluency, pronunciation, pausing, stress, and intonation.
Response length contributes some information because developing ideas is part of effective communication. But we also use features that distinguish original content from repeated material.
Repeating words, copying the question, or adding unnecessary language doesn't provide the same evidence of proficiency as developing a clear, relevant response. We also design our models so that no single area, such as vocabulary, dominates the final score. Our goal is to evaluate how effectively test takers communicate using English.
Can the DET's AI scoring models change while scoring tests?
No. Once we approve a scoring model for use, we lock it. It does not continue learning or adjusting itself while scoring test taker responses.
This is an important part of maintaining consistent scoring. We periodically update our scoring models, but every proposed update goes through a structured process of training, evaluation, fairness testing, and expert approval.
These updates can improve how accurately and consistently our models evaluate responses. For example, we might improve how a model handles unusual responses or identifies answers that don't address the question. The underlying aspects of English proficiency we're measuring remain the same.
By carefully evaluating each update before implementation, we can improve scoring performance while maintaining consistent standards.
What does responsible AI scoring look like in language assessment?
At the DET, we use AI and machine learning to evaluate complex speaking and writing skills at scale. Human expertise guides every stage of that process.
Our assessment scientists determine what the test measures. Expert raters provide the judgments we use to train and evaluate models. Our researchers test accuracy and fairness, monitor performance, and approve updates before they're introduced.
For institutions using DET scores in admissions decisions, this provides a clearer picture of how automated scoring works and the evidence behind it. For test takers, it means their speaking and writing responses are evaluated using a broad range of language features within a scoring system developed and monitored by assessment experts.
As AI becomes more common in education, understanding how these systems work will help students and institutions evaluate their use more confidently.
Our approach to automated scoring combines advanced technology with rigorous human oversight. We continue to invest in research, transparency, and fairness to ensure our scoring systems provide accurate, consistent evidence of English proficiency.