Recognition-Guided Generative Skeleton Imputation for Robust Action Recognition Under Severe Occlusion in Construction Scenarios

Abstract

Skeleton-based human action recognition can support digital construction and renovation monitoring by converting worker motions into structured information for safety and process management. However, workers are often occluded by tools, materials, furniture, and temporary structures, resulting in incomplete skeleton sequences and reduced recognition accuracy. In this study, we propose a recognition-guided generative skeleton imputation approach for robust action recognition under severe occlusion. Our method first evaluates an incomplete skeleton and applies imputation only to samples with uncertain recognition results. For these samples, a pretrained OmniControl model generates multiple plausible motion candidates from the observed skeleton information and action descriptions. The candidate most consistent with the observed motion is selected and used only to fill missing regions while preserving reliable joints, and the completed skeleton is then re-evaluated by the same recognition model. Experiments on 565 collected indoor renovation clips and the NW-UCLA dataset, using four recognition models and nine occlusion conditions, demonstrated overall improvements in recognition robustness, with average accuracy increasing from 61.01% to 65.00% on the collected dataset and from 42.63% to 50.31% on NW-UCLA. These results demonstrate the potential of selective generative imputation for improving action recognition under incomplete skeletal observations.