YOLO-CNN-Based Multimodal System for Real-Time Structural and Bolt Analysis on a Semiautonomous Robot

Abstract

Abstract This study introduces the Remote Automated Multi-Bolt Inspection Robot (RAMBIRobot), a novel system integrating machine vision and acoustic analysis for efficient, autonomous bolt inspection. Conventional bolt inspection methods are labor-intensive, prone to human error, and unsuitable for hazardous environments. RAMBIRobot addresses these limitations by combining the YOLOv5s visual recognition model, audio-based detection via Mel-Frequency Cepstral Coefficients (MFCCs), and a two-dimensional convolutional neural network (2D CNN) under the robot operating system (ROS) framework. The system features a robotic arm equipped with a depth camera for 3D bolt localization and a tapping device for acoustic analysis, providing robust and automated inspection capabilities. Experimental validation demonstrated that the audio recognition module achieved an accuracy of 79.3% at a 90° tapping angle, with loosened bolt detection precision of 90.2%. The YOLOv5s model achieved an accuracy of 98.9% under normal lighting and 93.5% in low-light conditions, confirming its reliability across diverse environments. Integrated system tests revealed an overall detection accuracy of 75.0%, highlighting the system’s efficiency in reducing dependency on manual labor while ensuring inspection precision. RAMBIRobot offers a cost-effective, scalable solution for industrial applications, enhancing structural health monitoring through automation. Compared to existing robotic inspection systems, RAMBIRobot uniquely integrates visual localization with acoustic analysis for bolt-specific looseness detection on a mobile platform. Future research will focus on optimizing environmental adaptability and exploring additional inspection scenarios.