Practical reliability-aware temporal state correction for depth-assisted RGB-skeleton action recognition

Abstract

RGB-D action recognition benefits from skeleton motion, RGB appearance, and depth geometry, yet auxiliary modalities are not equally helpful for every sample. We present a reliability-aware temporal state correction method that treats the skeleton representation as the primary temporal state and RGB and local depth features as corrective evidence. A low-cost state-unreliability proxy combines class-normalized prediction entropy with temporal variation between unit-normalized hidden states; both components are dimensionless and bounded before fusion. A monotonic sigmoid gate then controls sample-wise RGB correction, while skeleton-guided local depth features provide a subordinate geometric residual. On NTU RGB+D 60, the full model achieves 93.7%/96.0% under the cross-subject/cross-view protocols; on NTU RGB+D 120, it achieves 90.2%/91.8% under the cross-subject/cross-setup protocols. Under a strictly matched NTU RGB+D 60 cross-subject decomposition, reliability-conditioned RGB correction improves naive integration from 91.8% to 93.1%, and local depth further raises accuracy to 93.7%. The combined proxy obtains an error-detection AUROC of 0.72 on the held-out validation split. We interpret this score only as an empirical conditioning proxy, not as Bayesian or calibrated uncertainty.