Trans-Percept: robust perceptive locomotion in the wild via spatiotemporal causal transformers

Abstract

The integration of exteroceptive perception with proprioceptive control has revolutionized legged robot locomotion, enabling navigation through complex, non-flat terrains. Miki et al. (2022) established a seminal paradigm by fusing vision and proprioception using Recurrent Neural Networks (RNNs) and Belief States. However, RNN-based architectures suffer from inherent limitations in modeling long-horizon dependencies and reacting to high-frequency burst noise in sensor data. In this paper, we propose Trans-Percept , a novel end-to-end reinforcement learning architecture based on Causal Spatiotemporal Transformers. Unlike recursive belief updates, Trans-Percept processes a sliding window of raw multimodal history, utilizing self-attention mechanisms to dynamically weigh the reliability of visual data against proprioceptive feedback in real-time. We introduce a “Modality Confidence” masking strategy during teacher-student training that enforces robust behavior during sensor occlusion. We benchmark our method against four distinct baselines: (1) Blind Locomotion (RMA), (2) Recurrent Belief State (Miki et al.), (3) Conventional Elevation Mapping, and (4) A Naive Vision Transformer. Extensive experiments in high-fidelity simulation and on the Unitree Go2 quadruped demonstrate that Trans-Percept outperforms the state-of-the-art Recurrent baseline by 18.4% in success rate on “visually deceptive” terrains and reduces recovery time from unexpected external perturbations by 0.5s . The proposed method bridges the gap between reactive blind walking and predictive visual planning, offering a unified solution for wild environments.