Hierarchical Manipulation Skill Learning for Long-Horizon Robotic Manipulation Based on Residual Finite Scalar Quantization

Abstract

Leveraging discrete sequence prediction for manipulation skill learning has demonstrated immense potential for complex long-horizon tasks. However, guaranteeing precise and robust execution under complex scenarios remains a core challenge for efficient autonomous robotic operation. This demands execution policies capable of generating precise and efficient action sequences based on perceptual feedback. To address the loss of fine-grained action details induced by action discretization and the difficulty in preserving long-horizon temporal dependencies in long-horizon tasks, this work proposed a hierarchical manipulation skill learning method based on residual finite scalar quantization (RFSQ). At the action representation level, a robust RFSQ with an adaptive linear scaling mechanism was introduced. It effectively mitigated residual decay in multi-level action quantization and realized accurate discrete reconstruction of continuous action sequences. At the action sequence generation level, a hybrid hierarchical architecture integrating Mamba2 and Transformer was constructed. Mamba2 efficiently captured temporal dependencies among actions, while Transformer precisely modeled fine-grained residual dependencies across action quantization hierarchies. A multimodal perception fusion module was further incorporated to strengthen instruction-semantic alignment on language-instructed tasks. A series of experiments were conducted on the LIBERO and Meta-World simulation environments. Experimental results demonstrated that the proposed method achieved outstanding performance in multi-task and long-horizon task learning.