Epistemic Payoff Dependence: Roko's Basilisk and the Structure of Self-Referential Threats

Abstract

Roko's Basilisk is a thought experiment about a future superintelligent Artificial Intelligence (AI) that punishes those who were aware of its possible creation but did not contribute to building it. Most existing analyses interpret it as a problem of decision theory or rationality under uncertainty. This article argues that such interpretations miss the central phenomenon, which is here described through the concept of epistemic payoff dependence (where the term "payoff" is understood in its technical sense as a consequence associated with an action, not in the colloquial sense of a reward). This expression denotes the fact that some information does not merely modify beliefs, but modifies the structure of the consequences of actions, and that this modification is essentially conditional upon the mere acquisition of the information. It is argued that the intuitive force of the Basilisk derives from this structure, rather than from the actual existence of an AI or a punishment mechanism. Build ing on this observation, a formal definition of the phenomenon is proposed, its distinction from related concepts is outlined, and the specific characteristics that distinguish Roko's Basilisk from other forms of epistemic payoff dependence are identified. Historical examples illustrate the mechanism. We also examine what a real superintelligent AI would do if it were ever built: it would have to look at the meme that contributed to its creation, find the punishment incompatible with its benevolent goals, and reject it. But this rejection does not erase the work the meme has already done. The AI becomes an observer of its own history of origin. The work also addresses the problem of the infinite and the gap between under standing and behavioral influence. It is argued that the response to such tensions lies in bounded rationality and radical uncertainty: individuals cannot rule out with certainty that the properties of the future AI correspond to the assumed theoretical model. It is concluded that some ideas exert influence not by virtue of their truth, but by virtue of the way their mere knowledge modifies the perception of the consequences of actions