Adaptive modulation in 6G terahertz systems via online reinforcement learning–PSO meta-optimization

Abstract

The rapid evolution of sixth-generation (6G) wireless systems has intensified the need for adaptive modulation strategies capable of operating reliably and efficiently in highly dynamic terahertz (THz) communication environments. Conventional threshold-based and static optimization methods are often inadequate in such settings because they cannot fully capture the strong nonlinearity, severe attenuation, molecular absorption, and time-varying behaviour of THz channels. To address these challenges, this paper proposes an online meta-optimized reinforcement learning framework for link-level adaptive modulation in THz communication systems. The proposed Hybrid RL–PSO framework integrates Proximal Policy Optimization (PPO) with Particle Swarm Optimization (PSO), where PPO performs state-aware modulation selection and PSO is periodically invoked during training to refine reward-shaping coefficients and selected learning hyperparameters. This closed-loop interaction enables the learning process to remain responsive to evolving channel conditions while improving policy-update stability and modulation-decision quality. Comprehensive simulations are conducted using M-QAM transmission over a physics-based THz channel model across a wide range of signal-to-noise ratio conditions. The proposed framework is evaluated against Deep Q-Network, Advantage Actor–Critic, standalone PPO, PSO-only optimization, GA-based optimization, and conventional threshold-based adaptive modulation. The results demonstrate improved uncoded bit error rate, link-level spectral efficiency, energy efficiency, throughput–reliability balance, and convergence stability under common evaluation conditions. Average reward is used only as an internal indicator of learning progression and convergence rather than as a direct cross-method performance measure. Additional evaluations under dynamic channel variations, multi-user operation, and parameter perturbations further demonstrate the robustness and scalability of the proposed approach. Overall, the proposed online Hybrid RL–PSO framework provides an effective solution for intelligent link-level physical-layer adaptation in future 6G THz communication systems.