In the dynamic landscape of test security, Duolingo is uniquely poised to redefine the discourse and influence the strategies that are adopted industry-wide. At Duolingo, AI isn't just a tool for us; it's also a source of some big questions. How will it change the role of test professionals? Will students eventually use AI tools like ChatGPT in exams?
We were eager to discuss these issues and share our work with colleagues from across the security industry, at the recent Conference on Test Security (COTS) at Arizona State University!
AI is changing the game
A major topic of discussion at the conferencewas the potential disruption that AI can cause. Questions were raised about the role of AI in the future of test professionals and the potential ubiquity of AI tools in education.
During our keynote debate, experts discussed whether or not remote secure high-stakes testing would even be possible in the next few years as AI’s rapid advancements could create a landscape where testers find it easy to cheat, while test providers find it increasingly difficult to identify cheaters. These considerations are shaping our thinking as we navigate this new terrain.
But when it comes to preventative security, it's not just individual students anymore; we're seeing entire "tutoring" and "test prep" websites dedicated to cheating. The industry agrees that we need to take a stand against these sites and potentially push for legislative changes to better combat cheating rings and ID fraud.
Of course, it wasn't all hard work at the conference. We also had some time to bask in the recognition of our peers. Jim Wollack from UW-Madison gave us a shout-out, acknowledging the innovative work we're doing. It's always nice to know our efforts are being noticed!
Duolingo is ahead of the curve
It was clear that everyone is trying to figure out how to integrate AI into their operations. Fortunately for us, we're already ahead. As a digital first high-stakes test, we have been following AI's rapid advancements since we developed our test and have designed and continue to design many features based on this.
Take item generation, for example. Our vast "item ocean," a step beyond the traditional item pool, allows us to focus our security efforts beyond mere item theft. This means we can concentrate our efforts on innovating new solutions to other important test security problems, rather than fretting over item theft.
In our coordinated symposium, we presented the DET's novel approach to test security – a human-in-the-loop system. This system integrates AI and human decision-making to stay ahead in the security arms race. AI's capabilities are used to enhance human decision-making, while human expertise is used to develop more fair and effective AI systems.
In a panel presentation, members of our security and proctoring teams delved into the benefits and challenges of keeping test security in-house with vertical integration. This approach unifies key functions such as proctoring, item creation, and test security under one roof, ensuring a secure test administration. The panelists also discussed potential hurdles and strategies for effectively navigating them, providing attendees with actionable insights for their own test administration contexts.
Securing the future
It's clear that the Duolingo English Test is not just part of the conversation on test security – we're leading it. Our time at COTS affirmed our unique positioning, and provided our proctoring and security teams with insights to inspire our work moving forward. Stay tuned for more updates from us!