MARMOT: masked autoencoder for modeling transient imaging

Abstract

Abstract Pretrained models have recently emerged as effective priors for a wide range of vision and computational imaging problems. Transient measurements have emerged as a new imaging modality, in which a time-resolved sensor records photon counts over time bins. In confocal non-line-of-sight (NLOS) imaging, a hidden scene can only be observed through multi-bounce transient measurements, making reconstruction from sparse scans severely ill-posed. Existing NLOS methods are often task-specific and typically rely on dense measurements or dedicated supervision for each downstream application. We present MARMOT, a masked autoencoder for modeling transient imaging to facilitate NLOS applications in a self-supervised manner. Given a subset of transient histograms sampled on the relay wall, MARMOT encodes the visible measurements with a Transformer encoder and predicts the missing transients with a lightweight Transformer decoder. This design learns a reusable prior over transient measurements while naturally supporting sparse, irregular, and non-uniform scanning patterns. To enable large-scale pretraining, we build TransVerse, a synthetic dataset of one million confocal NLOS transients rendered from 500,000 Objaverse objects. We evaluate MARMOT in two complementary ways. First, the recovered dense transients can be used for transient completion and NLOS reconstruction from sparse scans. Second, the pretrained encoder can be transferred to downstream visual inference tasks, including classification, albedo estimation, and depth estimation. Across synthetic and real-measured datasets, MARMOT achieves competitive performance and strong robustness under high masking ratios, indicating that large-scale self-supervised pretraining provides a practical prior for transient imaging.