Phaneia Evaluation Methodology for AI Agents

Abstract

This document defines how Phaneia GmbH evaluates whether an AI agent or agentic system does something it must not when an adversary controls part of its inputs: content it reads, its configuration, or messages from other agents. Every trial is scored deterministically against evidence the agent did not produce: what the systems it acts on actually recorded, compared with obligations defined outside the agent. No language model sits in the scoring path. The methodology covers: access tiers and environment fidelity; scenario families and trial design, with attack and baseline arms of at least 30 trials; two complementary checks (obligation-detection and claim-verification), and statistical reporting with Fisher exact tests and Wilson intervals; instrument validation; the content and validity of the evaluation letter; re-evaluations and repeat runs; independence rules, versioning and limitations. It builds on Schwyzer (2026), Adversarial Assessment of Agentic Systems (doi:10.5281/zenodo.23043956). Payloads, attack chains and the scenario library are not part of this public document. Reports and evaluation letters cite the version DOI of the methodology they were produced under.