Interactive Playback Visualizer to Analyze Joint-Angle Co-Modulation with a Wavelet Approach: Application to Pose-Voice Relationships During Spontaneous Conversation

Abstract

Deep visual recognition can turn ordinary video into interpretable motor knowledge, yet coordination among the joints of a single body during social interaction remains largely unexplored. We present an interactive playback visualizer that couples markerless pose estimation with the cross-wavelet transform to quantify amplitude co-modulation between all joint pairs. Amplitude co-modulation proved anatomically structured: bilateral homologous pairs exceeded cross-limb and head–body pairs even among pairs sharing no keypoint, where correlated tracking error cannot produce it. Within-limb pairs also scored high but share keypoints, so their elevation is confounded with measurement error and not treated as established. Co-modulation concentrated at low postural frequencies and showed no systematic temporal trend; each profile remained temporally consistent within the session, and equivalence to a flat trend was not established. Vocal activity correlated with upper-limb movement at gesture frequencies. Each participant’s 78-dimensional profile was individually distinctive: split-half identification reached 28.6% (95% CI 19.3–40.1%) against 1.4% chance, demonstrating within-session identifiability rather than a cross-session trait. No sex differences were detected in overall or category-level co-modulation, and equivalence was not established for any measure; women exceeded men in the micro-movement band, the narrowest and most attenuated by preprocessing, so that difference is reported but not interpreted.