Sentinel-Augmented Reinforcement Learning with One-Step Ahead Predictive Safety Filtering

Abstract

Reinforcement learning (RL) consists of a general trial-and-error learning structure in which an agent, based on observations, receives rewards or punishments according to the action taken. This approach has proven to be efficient in many fields of application, such as video games, board games and even autonomous vehicles. But it is precisely in sensitive application areas, such as autonomous vehicles or other applications that require a high safety margin and need to eliminate errors, that the study of sentinel-assisted reinforcement learning was applied. It acts as a multi-agent reinforcement learning system with two collaborating agents. The first is the sentinel agent (teacher) that explores first, evaluates outcomes, and produces advice signals (positive / neutral / negative) to the main agent. The second one is the main agent (student) that receives these signals and adjusts its policy faster, minimizing trial and error. The research tries to mimic reinforcement learning from auxiliary feedback, similar to teacher-student Q-learning or meta-guided reinforcement learning. In Sentinel-Assisted Reinforcement Learning, the agent seeks to maximize the rewards, but your integrity during the execution of the program has priority over the positive rewards.