Bridging the gap: addressing measurement bias and integrating lexicons to generative AI

Abstract

Abstract Given the widespread application of lexicon-based computer-aided text analysis (CATA), we present several overlooked issues in current research practice. We examine the implications of arbitrary lexicon selection and the improper use of difference scores (positive minus negative) to measure overall sentiment. We provide detailed guidelines to help researchers avoid empirical bias that can lead to inconsistent results in lexicon-based CATA. Specifically, we recommend assessing each lexicon’s ability to capture sentiment by comparing coverage rates and outline when the use of difference sentiment score is appropriate. We also show how lexicons serve as contextual anchors in the use of generative AI (GenAI) through few-shot prompting. Using consumer review data, we replicate prior research on perceived review usefulness across widely adopted lexicons to demonstrate our guidelines. Our findings highlight that arbitrary use of lexicons may produce biased and inconsistent results across lexicons, limiting generalizability, and support complementary use of GenAI and lexicons.