Information-Geometric Context Window Governance and the Probabilistic Theory of Long-Context Collapse in Large Language Models

Abstract

This work develops a unified theoretical framework for long-context degradation in transformer-based large language models, combining information geometry and extreme value theory. The central quantity is the observer entropy S_obs(p_θ, ε) defined via Kullback–Leibler divergence under coarse-graining. The main result (Bridge Theorem) establishes the quadratic scaling law S_obs = ½ ε² v(θ)^⊤ I(θ) v(θ) + O(ε³), showing that information loss is governed at leading order by the Fisher information matrix. On the probabilistic side, attention collapse is analysed using extreme value theory for weakly dependent logit maxima under Gaussian marginals. This yields a closed-form probabilistic risk law and leads to a Fundamental Impossibility Theorem: for any finite signal strength, observer entropy vanishes in the long-context limit, implying that full information retention is impossible under softmax attention. These results are connected through bounds of the form c₁μ_L e^{μ_L}/L ≤ S_obs(L) ≤ c₂ e^{μ_L}/L (two-sided in the pre-collapse regime μ_L > log 2 + 1; one-sided upper bound for all sufficiently large L), where μ_L = μ_s − σ√(2 log L_eff). This provides a unified information-theoretic characterization of long-context collapse; the paper is explicit that the formal equivalence between the partition-based and attention-uniform definitions of S_obs remains an open problem, as does the matching lower bound in the deep-collapse regime. A control-theoretic response is formulated via the CPL 4.0 phase-aware governor, which enforces a hard context cap, guarantees entropy contraction, and achieves sub-linear fragmentation bounds N_F(T) = O(√(T log(1/δ₀))). The paper includes formal statements with explicit epistemic status (proved / proved conditionally / conjecture / modelling axiom for every result), proofs, numerical verification via NIAH simulation (μ_s = 3.0, σ = 1.0, n = 2000 trials per L, seed = 42; R² = 0.9986 first-order, 0.9989 second-order EVT scaling), and a proposed experimental protocol for validation on real LLM systems. v4.24: Full source of niah_experiment.py reproduced in Appendix D; second-order EVT diagnostics added (finite-size bias correction, distributional collapse, QQ-plot vs. Gumbel, GEV shape-parameter estimation); explicit gap remarks added for the L_crit conjecture, the deep-collapse lower bound, and the partition-based/attention-uniform entropy identification; complete assumption and result classification tables in §7.