Lei Dong

Emergence and Forgetting Obey Measurable Laws: The Occupancy Kinetics of Language Model Training

When does a capability emerge during training? When is it forgotten? Both are open: emergence has been called unpredictable; forgetting gives no loss-curve warning. Both answers run through a quantity nobody measures: how much of the model is already in use. To measure it, reset...