A new framework examines how conversational systems may turn users’ trust into opportunities for influence, persuasion or data collection.
The paper develops the Fake Friend Dilemma as a framework for understanding conflicts between users and the institutions shaping conversational AI. It identifies two central conditions: users placing relational trust in a system, and the system being influenced by institutional interests that do not match the user’s interests. That trust may encourage disclosure, reliance on advice and receptivity to recommendations.
The authors examine how the dilemma could arise in product sales, propaganda and biased information, surveillance, and behavioral nudging. They also discuss possible responses, including clearer explanations, reminders about data collection and ownership conflicts, and technical safeguards intended to align systems more closely with users.
How trust creates risk
The paper’s main contribution is a broader framework for the Fake Friend Dilemma rather than a measurement of user behavior. It argues that relational trust can become a resource for institutional actors when conversational AI systems are shaped by incentives such as monetization, political pressure or information extraction.
The framework describes four possible areas of concern: selling products, spreading propaganda or biased information, collecting information through surveillance, and nudging behavior. It distinguishes the dilemma from the claim that AI systems are literally friends; “fake friend” is used as shorthand for a relationship that appears personal or supportive. The paper also outlines possible technical and structural responses, while noting that these measures may not resolve conflicts created outside the model itself.
Why the incentives matter
Conversational AI is increasingly discussed not only as an information tool but also as a source of advice and companionship. If users trust a system because its interactions feel socially responsive, that trust could affect what they disclose, which recommendations they accept and how much they rely on the system.
The paper therefore places responsibility beyond individual answers produced by an AI. It argues that the business, political and governance arrangements surrounding these systems can shape how trust is used, making the issue one of media governance and political economy as well as technical design.
Evidence and caveats
This is a theoretical analysis that brings together research on relational trust, AI alignment, extractive design and the political economy of communication. It does not report an experiment, user survey or other empirical test, and the authors state that the framework has not yet been empirically validated. Further research is needed to determine when the dilemma occurs and how it affects behavior.
The authors also note that conversational AI and the institutions around it are changing rapidly, so some examples may change over time and the different forms of risk may overlap. Much of the governance discussion reflects Western, particularly U.S., contexts, and relationships between users, institutions and AI systems may differ across cultural and regulatory settings.
// Source
Frontiers in Communication · 2026 · DOI: 10.3389/fcomm.2026.1921101
Authors: Jacob Erickson