SparseRoute: Structured Sparse Variable Routing for High-Dimensional Multivariate Time Series Forecasting

Abstract

Forecasting high-dimensional multivariate time series requires models to represent dependencies across variables without making variable interactions prohibitively expensive. Full attention over variable-level tokens has a pairwise cost quadratic in the number of variables; channel-independent methods avoid this cost but omit explicit cross-variable interactions. SparseRoute addresses this design choice by assigning variables to a small set of learned subspaces and applying attention to their aggregated representations. A vector-quantization routing module produces hard variable-to-subspace assignments, while a straight-through estimator supplies surrogate gradients during training. A dual-path decoder combines subspace predictions broadcast to their assigned variables with variable-level predictions from a shared multilayer perceptron. We evaluate the model with three seeds on Traffic, Electricity, Solar-Energy, PEMS04, and PEMS07. SparseRoute obtains the lowest mean MAE for Traffic at horizon 96 and the lowest mean MSE at horizon 192; Crossformer leads on several other dataset–horizon pairs. Relative to a dense variable-token model with matched tokenization and backbone, routing changes the pairwise attention factor from the square of the variable count to the square of the subspace count; this theoretical comparison does not establish an end-to-end resource advantage over the external baselines.