L. Ghiringhelli

Sampling Weights at Intermediate Temperatures is Optimal for Training Large Language Protein Models

Using a statistical mechanics framework, we investigate the parameter space of transformer models trained on protein sequence data. We sample the loss landscape at varying temperatures using Langevin dynamics to characterize the low-loss manifold and to understand the mechanisms...