[Xiaomi Open-Sources Video Dubbing Model ControlFoley, Giving Users Full Control Over Sound Design] According to monitoring by Beating, Xiaomi's large model application team has released and open-sourced the video sound effect generation framework ControlFoley. Traditionally, AI video dubbing primarily relied on models to infer sounds based on visuals, making it difficult for creators to precisely control sound styles. ControlFoley emphasizes 'controllability': it can generate sounds based on visuals, accept text descriptions, or reference audio, allowing creators to produce sounds according to their intentions. For example, changing a knocking sound to a 'metallic knocking sound' or using drum tones to match tennis hitting actions—this model can maintain audio-visual synchronization while adhering to the specified style. At its core, ControlFoley employs a spatiotemporal audio-video encoder based on a modified CAV-MAE and introduces a 'time-timbre decoupling' strategy, assigning sound timing to the video and sound style to the reference audio. In multi-task evaluations outlined in the paper, ControlFoley achieved open-source SOTA (state-of-the-art) levels in several standard video dubbing tests. Even when text instructions strongly conflict with visual content, the model can balance text adherence and timing synchronization. Compared to the commercial closed-source system Kling-Foley, ControlFoley demonstrates competitiveness in metrics such as semantic alignment, synchronization, and perceptual quality; however, it still falls short in certain KL divergence matching metrics in Kling-Audio-Eval and MovieGen-Audio-Bench. Currently, the project's technical report, code, model weights, and demo have all been made available. [Original Link]