Addressing Is Not Free: Factored Knowledge Storage and the Key-Discrimination Limit of Dense Models

Abstract

Dense language models store factual knowledge superposed in their weights, paying an addressing cost that is real but invisible: no line item in any capacity account says what it costs to find a fact. We present a pre-registered measurement study of that cost from both sides of an architectural divide. On one side, we build a Factored Knowledge Architecture (FKA): a small reasoning kernel, content-computed compositional keys, and a factorized substrate storing values as pointers into shared structure. Under an accounting frozen before any measurement — every bit required at inference, amortized where shared — FKA stores 3.15 bits per parameter (3.51 marginal) at 2 million entities and 8 million facts, with 99.9% addressability at that operating point and 99.8% retrieval on entities never seen during training at N = 2 ,000 (99.9% direct, 99.6% composed), exactly zero edit interference measured at N = 2,000, and zero seed variance at its operating point. Its addressing cost is explicit: 2.38 · log2 N bits per entity, priced and bounded. On the other side, we measure a dense transformer baseline rehabilitated four times to fight at full strength — literature-matched recipe, corrected data pipeline, fair tokenization, honest lexicon accounting. It stores at most 0.298 bits per parameter at any scale we ran, an order of magnitude below FKA’s measured point, because it collapses on key discrimination at under 0.19× of its bit capacity. Discriminable keys grow as P 0.224 [0.185, 0.291] across a measured decade, and the transition is run-stochastic: identically trained seeds differ by 60+ points in recall at indistinguishable loss. The apparent contradiction with published ∼2 bits/param results dissolves under two measured factors — key amortization ( 2.0×) and tokenization surface (1.17×) — which over-cover the observed 1.89× gap. Dense capacity claims, we conclude, are claims about regimes below an addressing wall that scaling does not remove; a two-point extrapolation of that wall to 2M keys spans 162 orders of magnitude , and we decline to make it. Every reported number carries its bracket, width, and provenance; every threshold that adjudicated a verdict carries the measurement that set it.\n\nThis deposit names internal commit 2ebcb4409ec35c3189cfc769f3e7d7b661def029 as the state of record (not publicly obtainable). The enclosed source archive is the curated export supporting this paper's claims; its MANIFEST.json maps every archived file to its content identity at that commit. Companion repository: https://github.com/Vexillon-ai/Athena-FKA. Enclosed code is MIT-licensed; this deposit's documents are CC-BY-4.0.