Unified driving maneuver detection and classification from monocular video

Abstract

Autonomous driving systems must read the surrounding environment continuously, and they must do more than classify maneuvers: they also have to localize when each maneuver starts and ends. Most commercial systems and academic baselines treat temporal action detection (TAD) and action classification as separate tasks. This fragmented pipeline blocks feature sharing and lets errors propagate. Existing Transformer-based models also have quadratic complexity O (N²) and rely on non-causal inference, which means high latency and reduced sensitivity on motion-sensitive categories such as curves and U-turns. We propose a unified architecture with staged training that supports maneuver classification and temporal localization at the same time, from a single RGB video stream. The framework combines a Selective State Space Model (Mamba) for linear-time causal temporal modeling (O(L)) with a dual-pathway feature fusion mechanism that pairs a primary visual representation with an image-derived ego-trajectory branch. The visual pathway extracts spatio-temporal appearance features from a Vision Transformer backbone; the auxiliary pathway predicts a compact latent motion descriptor from the same visual tokens. No external inertial or telemetry sensor is required at inference. The two pathways are fused by concatenation, which preserves the motion-sensitive signals that cross-attention tends to dilute. For temporal action detection, we use a four-stage pipeline: Mamba-based causal sequence processing, boundary-aware localization, proposal generation, and graph convolutional refinement. The selection mechanism of Mamba adapts to the input, keeping critical driving events while dropping irrelevant segments. We evaluate under standard offline metrics and a strict 2-second causal-buffer protocol. Under the causal-buffer protocol, our framework reaches the best average localization accuracy among all evaluated methods (84.1% average, beating the strongest baseline by 2.9 percentage points), with the largest per-class gain (+5.9 points on U-Turn); the advantage reproduces on the public Honda HDD benchmark. The Mamba-based detector runs at 24.5 FPS, about three times faster than a matched Transformer backbone, while keeping competitive offline detection performance. Ablation studies confirm that concatenation fusion outperforms cross-attention and weighted-sum alternatives, and that the Mamba backbone and the graph convolutional component each contribute to overall performance.