Error-Controlled Mixed-Precision Cholesky Factorization with MX Microscaling Formats

Abstract

Existing tile-centric mixed-precision Cholesky factorization, as used in recent Gordon Bell campaigns, is based on a heuristic that bounds only rounding contribution while ignoring underflow when classifying tiles for different IEEE floating point precisions. Direct application to new low-precision formats can produce numerically unstable outcomes. We introduce an error-controlled ladder strategy for tile-centric precision assignment that addresses the underflow challenge, extending mixed-precision Cholesky factorization with Open Compute Project (OCP) standardized Microscaling (MX) block-scaled formats. Proven numerically robust, our ladder bounds rounding and underflow errors at each candidate precision and selects the smallest one satisfying a target tolerance. On Matérn covariance matrices, existing approaches silently overshoot ε by more than an order of magnitude at loose tolerances where FP8 underflow dominates error, whereas our ladder approach better matches user-specified ε by relying on hardware-supported low-precision formats (MXFP8/MXFP4). Furthermore, with our proposed MXFP16 extension, our approach delivers up to 1.79× speedup over the same approach with just IEEE FP formats on NVIDIA GB200 GPU, while reducing memory by roughly 30% and preserving accuracy bounds violated by previous approaches. To our knowledge, this is the first study to evaluate MX formats (down to MXFP4) as working storage-and-compute precisions in dense matrix factorization.