Investigating the role of pragmatic visual signals in language emergence using communication games and agent-based models

Abstract

Abstract Studying the emergence of language has often relied on communication games and agent-based models, with a strong focus on referential linguistic signals, often overlooking pragmatic signals that are frequently expressed visually, such as facial signals. These signals are a crucial component of natural interaction, supporting repair, coordination, and negotiation of meaning. We outline existing language emergence experiments, highlighting findings from communication games and research on pragmatic facial signals in interaction, demonstrating their relevance to the emergence of communication systems. Furthermore, computational models of language emergence, such as agent-based models, largely overlook that communication is layered across multiple communication channels. We argue that incorporating pragmatic visual signals into these approaches can help to provide a more nuanced understanding of the processes underlying language emergence in natural, multimodal interaction. Following these insights, we propose two complementary directions for future research. First, communication game paradigms can be adapted to systematically analyze visual pragmatic signals, linking them to existing commonly used measures such as convergence, communicative success, and adaptation of linguistic forms. Second, agent-based models can be extended to multi-channel frameworks in which agents produce and interpret both referential and pragmatic signals, enabling investigation of how layered multimodal information shapes language emergence.