The Brand Fact Error Rate: Study Protocol, First Edition

Abstract

Pre-registered study protocol. This record contains no results. Commercial measurement of brand presence in AI assistants reports whether an engine mentions, cites or recommends a brand. This protocol measures a different property: whether the factual statements an engine makes about a brand match what the brand itself publishes. We are not aware of a public dataset that measures this at scale. The protocol fixes, before data collection: ground truth taken exclusively from each brand's own published pages, snapshotted with URL and date before querying; nine machine-checkable fact types (entry plan price, currency, tier count, free tier, HQ country, founding year, founder names, a named integration, a named market or language) plus one reserve; a sample of 150 to 200 European B2B SaaS companies with public pricing, stratified by stage across at least five countries; three runs per engine, brand and fact on non-consecutive days, with a stability band reported alongside every headline figure; a five-bucket scoring rubric (correct, stale, fabricated, hedged, absent); and reporting rules that aggregate results and never present a named brand unfavourably. Limitations: this is a protocol, not a result. Sample size, engine list and country coverage are targets. The results will be published as a separate record and linked from this one, together with the prompt templates, the scoring rubric and the brand selection criteria. The method is intended to repeat quarterly.