WeatherAgent-SAM: Text-Grounded Agentic Tool Selection for Robust SAM Segmentation

Abstract

The Segment Anything Model (SAM) provides a versatile promptable segmentation interface, but adverse weather and image degradation can obscure visual cues and substantially reduce its accuracy. Existing approaches mainly follow two directions to mitigate this issue. The first adapts SAM through fine-tuning or additional trainable modules, which requires model-side optimization. The second preprocesses degraded images before SAM inference, but such methods are generally optimized for perceptual restoration rather than downstream segmentation. We introduce WeatherAgent-SAM, an agentic framework that improves degraded-image segmentation through response-conditioned preprocessing selection while keeping SAM frozen. WeatherAgent-SAM takes SAM’s initial response and a spatial prompt as inputs and uses response-conditioned knowledge to determine which preprocessing method should be applied next. To realize this process, the framework comprises two key modules. First, Fine-Grained Response-Conditioned Knowledge Construction adopts a progressive coarse-to-fine strategy: it first categorizes SAM responses into recurring states and then explores how individual preprocessing methods affect segmentation within each state. This design enables the agent to distinguish methods that pursue similar enhancement objectives but produce different segmentation outcomes. Second, a learnable Mask Verifier estimates the quality of each resulting SAM mask without ground truth and provides feedback to the agent, allowing tool selection to proceed iteratively and stop when a satisfactory prediction is identified. Experiments demonstrate that WeatherAgent-SAM achieves results comparable to those of the evaluated baselines on the Robust-Seg dataset and outperforms SAM, RobustSAM, and GaRA-SAM-Syn on the ACDC dataset.