Estimating the threat of AI-agent responding across online survey platforms

Abstract

Recent research and advances in LLMs have led to widespread concern that AI agents could pose as human online survey-takers. However, it remains unclear how prevalent AI agents are on survey research platforms and how to effectively detect AI agents. We validated a series of AI detection tests that effectively separated verified-human participants from both prompted commercial and purpose-built AI agents. Using these tests, we collected surveys on seven online platforms and find wide variation in rates of respondents failing AI tests, ranging from 6 to 41% across platforms (compared to a 2.4% false-positive rate for in-person human participants). We further demonstrate that undetected AI agents can impact the results of online surveys. Our findings suggest that while there is an urgent need for AI detection tests and consistent, systematic monitoring of data quality on online platforms, some platforms currently appear to provide data with a low rate of AI agents.