Participants using GenAI in online studies: an overview and recommendations for researchers and reviewers

Abstract

Participants’ use of generative artificial intelligence (GenAI) to create responses in online studies can threaten data authenticity and research validity. This issue is particularly pressing in settings where participation is paid for. In response, researchers have begun to develop different approaches to prevent and detect GenAI use by participants. This comment illustrates the unintended consequences of researchers’ measures to prevent and detect participants’ GenAI use and aims to help researchers and reviewers better identify the potential consequences of these measures. Drawing on current literature, we identify seven different levels of participants’ GenAI use, differentiated by increasing degrees of technological sophistication and automation. These levels range on a continuum from the manual operation of GenAI by participants to the employment of automated GenAI agents. We discuss prevention and detection measures corresponding to these levels and how researchers’ measures can cause reactions from participants. Specifically, we suggest that extensive measures can trigger an adverse selection mechanism by driving out honest participants and inviting greater sophistication in GenAI use. Instead of engaging in a technological cat-and-mouse game with participants through deterrence, we argue that we should focus on the incentive structure to resolve the issue. We propose a combination of better payment and an opt-in verified participation mode, i.e., video proof, to reduce participants’ GenAI use in online studies and increase data quality. Building on this perspective, we provide practical recommendations for researchers and reviewers to improve research quality, maintain fairness, and save resources.