DyCoDiff: Dynamic Collaborative Diffusion for Text‐Image‐Audio‐to‐Video

Abstract

Text‐image‐audio‐to‐video (TIA2V) generation synthesizes videos whose semantics, appearance, and temporal dynamics are jointly controlled by text prompts, reference images, and audio signals. Existing TIA2V methods usually combine these modalities through separate spatial and temporal attention branches, limiting crossmodal interaction and audio‐guided motion alignment. This work proposes DyCoDiff , a dynamic collaborative diffusion framework for TIA2V generation. DyCoDiff introduces a m ultimodal condition fusion layer (MCFL) that fuses text, image, and audio conditions through joint self‐attention and text‐centric crossattention before temporal denoising. A gated residual strategy injects the fused condition into the temporal branch while preserving the original spatial text‐image condition. Collaborative losses are further designed in a shared multimodal embedding space, including video–text, video–image, and dynamic video–audio alignment terms. The audio–video loss compares temporal feature differences rather than static clip‐level embeddings, making it suitable for rhythm‐sensitive generation. Experiments on AudioSet‐Drums‐VAT, Landscape‐VAT, and URMP‐VAT show dataset‐dependent gains and trade‐offs in distribution quality, visual consistency, and temporal stability. On AudioSet‐Drums‐VAT, it reduces Fréchet Video Distance from 98.31 to 72.97, Fréchet Inception Distance from 1782.97 to 719.92, and temporal flicker from 163.43 to 54.73. The results also reveal dataset‐dependent trade‐offs, indicating that fine‐grained music‐performance videos require more conservative multimodal fusion.