Xiaomi Open-sources Controllable Video Sound Effect Generation Model ControlFoley

Xiaomi's Large Model Application Team open-sources the controllable video sound effect generation model ControlFoley, which uniformly supports three types of tasks: text-guided video dubbing, text-controlled video dubbing, and reference audio-controlled video dubbing. The model solves problems such as weak text control and difficult reference audio synchronization in existing solutions through joint visual encoding, time-timbre decoupling, and modality-robust training, achieving open-source SOTA performance on multiple benchmarks including VGGSound-Test.

March 10, 2016 ControlFoley is a unified and controllable multimodal video-to-audio (V2A) generation framework that enables precise control over generated audio using video, text, and reference audio.

Best audio quality (IS). Notably, the model's performance surpasses domain-specific benchmark models such as CondFoleyGen, demonstrating strong generalization capabilities. ControlFoley also shows competitiveness when compared with powerful proprietary closed-source systems like Kling-Foley...

2 days ago ControlFoley is Xiaomi's open-sourced controllable video sound effect generation model, capable of solving controllability challenges in the V2A field. The model uniformly supports three types of video dubbing tasks: text-guided, text-controlled, and reference audio-controlled, through the self-developed spatiotemporal audio-visual encoder CAV-MAE-ST...

June 10, 2026 ControlFoley achieves state-of-the-art performance across all benchmarks, consistently obtaining the highest CLAP scores and lowest DeSync. It also significantly improves...

This page provides an editorial summary based on publicly available information. It is not a republished article. Use the source link below for the original report.