Bayesian Ambiguity-Contraction-Based Adaptive Robust Markov Decision Processes for Adversarial Surveillance Missions

Abstract

Collaborative Combat Aircraft (CCAs) are envisioned to enable autonomous intelligence, surveillance, and reconnaissance (ISR) missions in contested environments, where adversaries may act strategically to deceive or evade detection. These missions pose challenges due to model uncertainty and the need for safe, real-time decision-making. Robust Markov decision processes (RMDPs) provide worst-case guarantees but are limited by static ambiguity sets that capture initial uncertainty without adapting to new observations. This paper presents an adaptive RMDP framework tailored to ISR missions with CCAs. We introduce a mission-specific formulation in which the CCA alternates between movement and sensing states. Adversarial uncertainty is represented by finite threat prototypes defining observation, exposure, and state–action transition models. Our approach incrementally refines policies by eliminating inconsistent threat models, allowing the CCA to move from worst-case protection toward more efficient mission policies while maintaining robustness. We establish fixed-point stability, agreement with nominal planning as credible sets identify the true node-specific threat prototypes, and a conservative expected value guarantee while those prototypes remain credible. Experiments under Gaussian and non-Gaussian threat models across diverse network topologies show higher mission rewards and fewer exposure events compared to nominal and static robust planners.