Seeing ahead through a state-of-the-art review of visual perception and predictive intelligence for collaborative robots

Abstract

Abstract The rate at which collaborative robots (cobots) are evolving from isolated, pre-programmed manipulators into collaborative partners that operate productively alongside humans, other robots, and machines is accelerating rapidly. Advances in computer vision is enabling cobots, even in dynamic and cluttered settings, to detect, recognise and assess both environment and actions. However, achieving higher detection accuracy and real-time localization in dynamic systems remains a challenge, as most existing approaches assume uncluttered backgrounds with limited scene complexity. Closing this gap is critical for both research and industrial deployment. This paper reviews the development of visual perception in cobots, surveys emerging computer vision algorithms, and examines a holistic perception pipeline in which raw vision signals, vision language models (VLMs), and augmented/virtual reality interfaces jointly power a unified sense-think-act loop. The review finds that cobots in high-mix, reconfigurable factories face serious challenges: occlusions in fields of view, fluctuating illumination, ambiguous backgrounds, few- or zero-shot object scenarios, high-speed motion, and computational problems in fusing heterogeneous data streams in real time. Cobots also need to manipulate damaged items, requiring vision algorithms that infer geometry from sparse cues and support adaptive grasping. Finally, the review identifies three complementary research directions to unlock plug-and-play cobots: (1) information aware sensor-fusion pipelines transmitting only credible voxels to reduce bandwidth while preserving sub-millimetre accuracy; (2) language conditioned planning and control networks mapping plain-language goals to smooth, jerk-bounded trajectories; and (3) pre-trained few-shot reward heads that score task progress without bespoke metrics, closing the reinforcement learning (RL) loop. Combined, these advances promise cobots that can be commissioned with relatively few demonstrations.