Hierarchical reinforcement learning with low-level agents and active curriculum for torpedo countermeasures

Abstract

Modern underwater warfare requires sophisticated torpedo countermeasure strategies to ensure mission success in complex threat environments. Previous hierarchical reinforcement learning (HRL) approaches show limited scenario adaptability, training efficiency, and structural flexibility. In this paper, we propose Multiple Specialized Low-Level Agent Hierarchical Reinforcement Learning (MSLA-HRL) for submarine torpedo countermeasures, featuring a 1:N architecture where a strategic high-level agent coordinates N specialized low-level agents. Each low-level agent independently develops expertise for specific threat scenarios, while the high-level agent performs strategic selection and decoy deployment planning. In order to train this hierarchical structure, we integrate Active Curriculum Learning (ACL) with three modules: tactical guideline-based Scoring, Adaptive Pacing, and continuous Score Update. Our ACL dynamically adapts to evolving agent capabilities, enabling stable convergence in sparse reward environments with non-stationary dynamics inherent in hierarchical learning. Experimental validation across 630 torpedo threat scenarios demonstrates that the complete MSLA-HRL framework with ACL achieves a 97% mission success rate, representing a 17.8%p improvement over our previous 1:1 hierarchical model. The results indicate improvements in training stability and operational flexibility. This work suggests that adaptive hierarchical systems with curriculum-based training can mitigate key limitations of reinforcement learning in underwater warfare scenarios.