Algebraic Geometric Empirical Process (AGEP) Theory of Deep Learning Dynamics

Abstract

Modern Transformer architectures have achieved outstanding empirical success, but their parameter spaces are fundamentally characterized by complex algebraic singularities, such as rank collapse in self-attention mechanisms, which invalidate classical regular statistical asymptotic theory. While Sumio Watanabe's Singular Learning Theory (SLT) provides a powerful static framework for Bayesian generalization via the resolution of singularities, the dynamical behavior of empirical probability processes undergoing gradient updates in the neighborhood of these singular loci remains unexplored. In this paper, we propose "Algebraic Geometric Empirical Process (AGEP) Theory of Deep Learning Dynamics" as a new unified mathematical paradigm to characterize the learning dynamics of Stochastic Gradient Descent (SGD) on singular schemes. By excluding the transcendental Softmax function from the self-attention mechanism and formulating the output as a homogeneous polynomial map, we enable the direct application of scheme theory. Under this formulation, we define the rank-1 singular locus of a 2x2 linear self-attention model as a closed subscheme defined by determinantal ideals. We then apply a monoidal blow-up (resolution of singularities) of the quadric cone xy - zw = 0 to mathematically prove that the uniform Donsker property of the empirical process is algebraically restored in local coordinate charts, justifying tight asymptotic weak convergence to a limit Gaussian process. Furthermore, we provide empirical verification of this algebraic-geometric transition using numerical simulations in PyTorch. We demonstrate that SGD trajectories are dynamically attracted to the singular subscheme, structurally coupling empirical loss minimization with the collapse of the determinant of weight matrices toward zero. This attraction causes a severe learning plateau accompanied by the spectral degeneration of the Fisher Information Matrix (FIM), which we characterize geometrically as a projection onto the exceptional divisor. This work represents the first bridge between pure scheme-theoretic structures and infinite-dimensional empirical process theory to dissect the dynamic learning trajectories of attention mechanisms, establishing a rigorous foundation for a complete geometric theory of deep learning dynamics.