Role-based multi-robot warehouse coordination with reinforcement learning assisted task selection

Abstract

We study decentralised coordination in large-scale RMFS, where capacity-limited fleets fulfil orders under stochastic demand, congestion, and real-time constraints. We formalise this problem as a role-structured decentralised partially observable Semi-Markov decision process that extends the semi-Markov multi-agent framework to include role-conditioned macro-actions with role-dependent eligibility constraints and the quorum role switching mechanism for density-driven, decentralised role transitions. Unlike generic macro-action initiation sets, our eligibility gates enforce physical feasibility by dynamically disabling roles based on instantaneous load, capacity, and shelf-assignment state, while the quorum role switching balances local task demand against agent supply without centralised coordination. To optimise within this structured action space, we introduce HSC-Net, a hybrid actor–critic architecture that combines role-specific heuristic candidate generation with convolutional spatial encoding and cross-attention-based candidate scoring, supported by a centralised graph-attention critic over the warehouse sector graph. Our framework enables asynchronous multi-hop relay without fixed infrastructure or agent rendezvous, decoupling intermediate shelf transport from direct delivery. Extensive simulations across different warehouse scales demonstrate significant improvements in throughput, completion time, and congestion balance against rule-based heuristics and value-based deep Q-learning baselines.