OmniTraj: Pre-training on heterogeneous data for adaptive and zero-shot human trajectory prediction

Abstract

Accurate trajectory prediction of vulnerable road users is a cornerstone of safe autonomous driving and intelligent transportation systems. While large-scale pre-training has advanced this field, achieving robust zero-shot generalization remains a critical challenge for real-world deployment, particularly when vehicles encounter unseen environments and heterogeneous sensor configurations (e.g., varying frame rates and observation horizons). In this work, we revisit zero-shot trajectory prediction from the perspective of distribution shifts and distinguish three transfer settings: temporal transfer, scene transfer, and joint scene–temporal transfer. Through systematic experiments, we show that temporal mismatch is a key source of failure in current pre-trained models. By isolating temporal configuration from dataset shift, we demonstrate that explicitly conditioning on temporal metadata provides a simple and highly effective solution. Building on this insight, we propose OmniTraj, a Transformer-based framework pre-trained on large-scale heterogeneous data with explicit temporal-aware design. OmniTraj is designed to handle omni-generalization in trajectory prediction, namely adaptability across temporal configuration and scene shifts. It achieves state-of-the-art zero-shot generalization under joint scene–temporal transfer, reducing prediction error by over 70%. Furthermore, it exhibits exceptional robustness in safety-critical edge cases with severely limited observations and maintains high few-shot data efficiency, paving the way for scalable, dataset-agnostic deployment in real-world autonomous systems. The code is publicly available: https://github.com/vita-epfl/omnitraj .