What if a language test could simulate real-time conversations? The DET’s latest item uses generative AI to do just that, allowing us to test language skills like never before—this is just one of the ways we use new technology to enhance how we measure language ability.
Understanding “interactional competence”
Imagine you’re a student and you’d like to ask your instructor for an extension on an assignment. Now imagine you want to invite a classmate to get together to study for an upcoming exam. You might use more formal language with your instructor, and be more casual with a fellow student… and that makes sense! These are two different requests, and two different power relationships, so your approach for each is going to be a little different.
In the world of language assessment, this is known as Interactional Competence: the ability to communicate in different social situations by interpreting the verbal and non-verbal cues of others and responding appropriately. Interactional competence draws on multiple skills, like collaboration, problem solving, and the ability to understand the social dynamics and expectations.
Because interactional competence is all about the ability to vary language and tone depending on the situation, it can be difficult to measure on a test. Tasks designed to assess it usually require test takers to listen to a conversation and answer comprehension questions about what they heard, because it wasn't possible to simulate real-time conversations on a test—until now.
Tests that talk back
The best way to test interactive skills is with interactive test items. While this isn’t really possible on traditional tests, the DET assessment scientists and machine learning engineers leverage AI to create interactive items that test interactional competence, and other skills, like never before.
Like Roleplay, our newest test item, Interactive Listening, puts test takers face-to-face with an animated conversation partner—played by our Duolingo world characters! First, the test taker is given a brief description of a common scenario they might find themselves in as a university student. They must then conduct a multi-turn conversation to achieve a certain communication goal—like following up with a professor for more information on a topic from class, or asking a friend to review a paper.
This item measures multiple language skills at once—which is how you use them in real life! By listening and responding to a conversation partner, test takers demonstrate comprehension and conversation skills, and writing a summary of the conversation afterwards demonstrates literacy and production.
Building character(s)
Interactive items like this wouldn’t be possible without a lot of AI. First, Generative AI helps us to create the scripts for each dynamic conversation, including the responses test takers can choose among as they move through the item.
We use more machine learning techniques and computer vision algorithms to create visemes: the distinct visual shapes that the mouth forms when producing specific speech sounds. Including visemes in our animations so that the characters are in sync with the audio makes for a smoother, more lifelike experience, but more than that, it actually aids comprehension: research suggests that the combination of auditory and visual cues helps test takers better understand what the animated characters are saying.
Then, we use Text-To-Speech (TTS) AI to convert the written scripts into spoken words. TTS uses algorithms to mimic human speech, taking into account rhythm, tone, and intonation to generate natural-sounding voices. Not only are generated voices easier to scale, they make for a consistent listening experience for test takers, eliminating any unfairness or variation that might come from using human voices.
AI at every step
AI comes into play in many ways throughout the DET, helping us generate and review interactive items, administer the test adaptively to every test taker, and even pilot new items that can further improve the DET’s Test Taker Experience, or TTX.
As we continue to evolve alongside the latest AI, we know it’s important to be transparent about the ways we use them in our test development process, which is why we’ve also developed responsible AI standards to ensure accountability and transparency. Embracing new technologies is central to our mission, and we’re proud to be leading the way!
To learn more about Interactive Listening, check out our paper in Frontiers in Artificial Intelligence!