Emergence and Forgetting Obey Measurable Laws: The Occupancy Kinetics of Language Model Training

Abstract

When does a capability emerge during training? When is it forgotten? Both are open: emergence has been called unpredictable; forgetting gives no loss-curve warning. Both answers run through a quantity nobody measures: how much of the model is already in use. To measure it, reset one small piece of the network to its untrained state; if anything the model has learned gets worse, that piece was in use. The fraction in use is the occupancy, F. It is not loss in disguise: five models at the same loss have F from 0.006 to 1.00. With F in hand, emergence obeys a formula: a capability needing K parts, fed by data at frequency p, arrives at t* ≈ 1/(νS·p^K·X(F)^K), with X ≈ 1 below F ≈ 0.55, collapsing beyond 0.7. Until all K parts are right, the assembly yields nothing and sends back no gradient—no optimizer steers toward it. Arrival is a K-way coincidence, and coincidences have arithmetic. Frozen-in-advance constants forecast emergence on six held-out models to 5% median error. Forgetting follows different quantities. It ignores occupancy; what matters is how widely the capability has spread—measured by knocking out top components singly versus jointly—and how deep each copy sits: its logit margin, which sets the erosion rate. Concentrated in one head, a capability dies in 43 steps of starved training; spread into backups, it lasts 3,283; re-concentrated late, 96; across independent trainings, matched spread with deeper margin buys 10×. The spread law forecast before measurement—and hit. Why do neural networks obey laws that sound like crystallization? Because any system in which progress needs K things right at once, driven by fluctuation, on finite capacity, with structures that decay unless maintained, obeys them—as do crystals and, point by point, transformer training. One class; membership, not analogy. All results use open suites (14M–1B); these gauges are built for industrial scale.