A collaborative perception framework for green jujube: from lightweight key point detection to posture-aware harvest analysis

Abstract

Abstract Automated harvesting is pivotal to smart agriculture, yet precisely localizing picking points in complex orchard environments remains a key challenge for robotic harvesting systems. To this end, this study proposes a collaborative perception framework integrating a lightweight pose estimation model termed RepViT-DSConv-ARC-PoseHead enhanced YOLO (RDA-YOLO) with Segment Anything Model 2 (SAM2). RDA-YOLO is the lightweight component responsible for real-time fruit detection and keypoint localization, while SAM2 is adopted for fruit contour segmentation and occlusion assessment. Based on the YOLOv11n-POSE architecture, we introduce Distribution Shift Convolution (DSConv) to reduce computational overhead and design the C3k2_RepViT module to enhance multi-scale feature fusion. Crucially, a specialized Adaptive Regression Coordination Pose Head (ARC-PoseHead) is developed to replace traditional prediction structures, employing log-scale polar regression and subpixel refinement to significantly improve coordinate perception of jujubes and key points under complex backgrounds. RDA-YOLO performs real-time detection and locates key points such as the pedicel-fruit transition zone to calculate optimal harvesting angles, while SAM2 performs fine-grained contour segmentation based on detection boxes to determine occlusion status and establish harvesting priority. Experimental results show that RDA-YOLO achieves 93.9% precision and 94.9% mAP@50 for green jujube detection, while for key point detection, it attains a superior precision of 90.4% and a mAP@50 of 83.6%. Notably, the model achieves a lightweight design by reducing parameters to 2.490 million and computational load to 5.7 GFLOPs. Deployment feasibility was verified on the field robot control computer and a robotic platform using jujube fruit models, demonstrating the potential of the proposed framework for practical harvesting applications.