Density Modeling of Sequences: A Path-First Theory of Autoregression, Masked Diffusion, Adaptive Insertion, and the Price of Order

Abstract

We develop a path-first theory of discrete sequence generation. A generator first defines a probability law on chronology-augmented construction paths; the terminal sequence law is the pushforward of that path law. The full geometric signature faithfully encodes every finite action trajectory after clock augmentation, whereas truncation loses high-order chronology and expectation passes from a sample path to moment coordinates of a law. We prove the corresponding covariance defect: the expected signature is group-like only when the construction path is deterministic. On a bounded finite trajectory space, the full expected signature determines the complete path law. Consequently, two decoders may have the same terminal distribution while remaining distinguishable at the path-law level. Within this framework, an exchangeable absorbing masked diffusion trains the same compatible conditionals as uniform random-order autoregression. After a change from diffusion time to retention probability, its continuous-time loss is exactly the expected autoregressive negative log-likelihood over uniform permutations. A common masking schedule changes the physical clock but not the rank-ordered construction path. Positive context reweighting preserves the unrestricted population optimum, while finite-capacity projection, optimization, discretization, and parallel sampling need not be invariant. The discrepancy from conditionally independent block prediction is exactly conditional total correlation. Any non-anticipating adaptive reveal policy preserves the target terminal law under exact conditionals, and its approximation error is bounded by relative entropy accumulated on trajectory space. For variable-length insertion, generation order is a learned control and termination is endogenous. We prove almost-sure termination under a uniform stop bound, the trajectory--permutation bijection, the exact permutation-marginalized likelihood, the variational lower bound and its gap, Rao--Blackwell variance reduction, and universal representation of laws on finite sequences. The resulting separation among path law, terminal law, content conditionals, order policy, transition rule, and training measure gives a precise common foundation for autoregression, masked diffusion, parallel denoising, and insertion-based planning.