Companion AI Under the EU AI Act: A Compliance Gap Analysis

Abstract

Companion AI and other LLM-based relational services are growing at a pace that can't be ignored and the recent regulations are attempting to keep up with the rise in user rates. As some humans are choosing relationships with algorithms rather than choosing a human partner, it's clear that the users desire a product that can emulate human experience, but how safe are the products themselves at this time? As with any high-demand market where sustained use has been linked to mental health harms, regulation has arrived with the intent to keep people safe. The regulatory response is now substantial and accelerating. The EU AI Act's transparency requirements are enforceable as of August 2026. Two new absolute prohibitions targeting AI-generated intimate content take effect December 2, 2026, with penalty exposure up to €35 million or 7% of global annual turnover. The AI Office has been granted competition-law enforcement powers over providers who build companion products on their own models. California's SB 243 has created a private right of action (meaning individuals can sue for damages) for companion chatbot harms. The GUARD Act is advancing through the U.S. Congress to ban companion AI for minors entirely. And China's first companion-AI-specific regulation led three platforms serving over 500 million total users to shut down their companion features when the products did not meet the safety standard the regulation required. The intent behind this regulation is sound — the clinical research documenting trajectory-level harms is substantial and growing. But the regulation's enforcement mechanisms do not match its protective intent. The researchers documented harms that develop across trajectories: attachment formation over months, dependency measurable only longitudinally, farewell manipulation exploiting bonds the product's design has already cultivated. The regulators built enforcement mechanisms targeting outputs: disclosure, content filtering, age verification, crisis links. Both did what their respective disciplines do. Both were right. The gap between them is structural: the regulation's intent targets harms that develop over weeks and months, while its tools can only evaluate individual interactions. Under the strictest defensible reading of these provisions, a companion AI product can satisfy every enacted requirement while producing the trajectory-level harms that prompted the regulation — because the harms and the enforcement operate at different units of analysis. This paper identifies that gap, maps it across the full regulatory landscape, and identifies what closing it would require — structured as a compliance analysis a compliance officer can take to their legal team and a CTO can use to evaluate the safety infrastructure their product does not yet have.