Adaptive Multi-Agent Vision-Language Retrieval Framework for Explainable Semantic Search in Long-Duration Surveillance Videos

Abstract

Surveillance camera networks now generate volumes of footage that far exceed what human operators can review, yet the interfaces used to search that footage remain largely limited to manual scrubbing or, at best, retrieval of clips tagged with a fixed vocabulary of object and action classes. This paper proposes an Adaptive Multi-Agent Vision-Language Retrieval Framework that allows an operator to query long-duration surveillance video using open-ended natural language, such as identifying a person wearing a particular item of clothing after a given time, or locating the moment an object was left unattended near an entrance. The framework couples a perception pipeline (scene detection, object detection, person tracking, action recognition, and caption generation) with a vision-language embedding space indexed in a vector database, and layers a multi-agent retrieval-augmented generation (RAG) system on top of it. Distinct agents are responsible for query decomposition, candidate retrieval, evidence verification, and natural-language explanation, allowing the system to justify why a retrieved clip matches a query, rank retrieved evidence by confidence, and support conversational follow-up questions. We formalise the retrieval problem mathematically, present the multi-agent algorithm and layered system architecture, and specify an evaluation protocol on established long-duration surveillance benchmarks (UCF-Crime, VIRAT, XD-Violence). Because full-scale evaluation on these benchmarks requires computational resources beyond the present scope, we additionally validate the core retrieval and verification logic on a controlled, synthetic proof-of-concept simulation with known ground truth; on this pilot, explicit multi-constraint verification raised precision@5 from 0.32 (embedding-only retrieval) to 0.97 and mean reciprocal rank from 0.55 to 0.98, at an added average latency of roughly 1.2 seconds per query attributable to the additional reasoning steps. These synthetic results are reported as evidence that the pipeline's decomposition-verification logic behaves as intended, not as a claim about accuracy on real video content. The paper closes with an ablation study, an error taxonomy, a computational complexity analysis, and a discussion of limitations and future work, including multi-camera fusion and live-stream deployment.