A Physics‐Guided Feature Selection Machine Learning Approach for Cloud Probability Retrieval and Detection From TEMPO Imagery

Abstract

Abstract Accurate cloud detection is critical for reliable retrievals of aerosols and trace gases from remote‐sensing measurements. TEMPO provides unprecedented hourly observations over North America, but radiometric calibration biases, thousands of spectral channels, and limited spectral coverage make cloud detection challenging. Here, we develop a physics‐guided feature selection machine learning (PGML) framework that integrates radiative transfer simulations with a LightGBM architecture to detect clouds from TEMPO hyperspectral observations. Our approach starts with radiative transfer simulations to identify the spectral channels most sensitive to clouds among the 2,056 available. An explainable AI method is then applied to rank feature importance, further refining model inputs to 25 ultraviolet and visible channels plus geometric angles, improving interpretability and computational efficiency. Using mature GOES‐R cloud masks as reference labels, the model is trained and optimized with a cross‐entropy loss function to predict cloud probability. The approach effectively captures diurnal variations in both cloud fraction and spatial structures. The resulting cloud masks show high classification performance against the reference masks, with a balanced overall accuracy (BOA) of 84.43%, outperforming the Level 1B cloud mask (73.86%), which tends to underestimate cloud cover. Independent validation with AERONET observations further confirms this improvement (BOA: 78.54% vs. 54.88%). Moreover, the framework generalizes well to GEMS, achieving BOA scores of 80.38%, 71.98%, and 77.42% against Himawari cloud masks, AERONET, and CALIPSO observations, respectively. These results highlight the framework's strong adaptability and transferability, as well as its potential to enhance atmospheric composition monitoring with the next generation of hyperspectral environmental satellites.