Real-Time Physical-to-Virtual Scene Reconstruction for Dynamic Manufacturing Systems via Semantic—Spatial Mapping

Abstract

Real-time reconstruction of virtual scenes in response to changing physical conditions remains a challenge in manufacturing digital-twin systems. During manufacturing operations, changes in object positions, orientations, and spatial arrangements require the virtual scene to be updated to reflect the evolving physical environment. Here, we present a physical-scene-driven semantic–spatial mapping framework to support real-time scene reconstruction in dynamic manufacturing systems. The framework integrates visual object detection, point-cloud-based spatial estimation, digital asset retrieval, and virtual scene updating. A lightweight detector identifies physical objects and provides semantic information for spatial processing, while object-level point clouds are used to estimate their positions and orientations. The resulting semantic and spatial information is associated with predefined computer-aided design (CAD) models and used to update corresponding virtual objects in a Unity environment. To improve the efficiency of semantic perception, we develop SF-Lite-YOLOv8n by integrating C2f_Star, Spectral-SPPF, and Detect_Efficient. On the MVTec ITODD dataset, the detector achieves an mAP@0.5 of 0.940 with 1.98 million parameters and 6.2 GFLOPs, at a measured inference speed of 83 frames per second. In experiments involving 20 self-collected manufacturing scenes, the integrated perception and mapping pipeline achieves mean position and orientation errors of 2.7 mm and 1.9°, respectively, with a processing latency of 146 ms, excluding Unity rendering. These results demonstrate the potential of combining lightweight semantic perception and point-cloud-based spatial estimation to support the reconstruction and updating of virtual manufacturing scenes.