MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation
CoRR(2024)
Abstract
Foley audio, critical for enhancing the immersive experience in multimedia
content, faces significant challenges in the AI-generated content (AIGC)
landscape. Despite advancements in AIGC technologies for text and image
generation, the foley audio dubbing remains rudimentary due to difficulties in
cross-modal scene matching and content correlation. Current text-to-audio
technology, which relies on detailed and acoustically relevant textual
descriptions, falls short in practical video dubbing applications. Existing
datasets like AudioSet, AudioCaps, Clotho, Sound-of-Story, and WavCaps do not
fully meet the requirements for real-world foley audio dubbing task. To address
this, we introduce the Multi-modal Image and Narrative Text Dubbing Dataset
(MINT), designed to enhance mainstream dubbing tasks such as literary story
audiobooks dubbing, image/silent video dubbing. Besides, to address the
limitations of existing TTA technology in understanding and planning complex
prompts, a Foley Audio Content Planning, Generation, and Alignment (CPGA)
framework is proposed, which includes a content planning module leveraging
large language models for complex multi-modal prompts comprehension.
Additionally, the training process is optimized using Proximal Policy
Optimization based reinforcement learning, significantly improving the
alignment and auditory realism of generated foley audio. Experimental results
demonstrate that our approach significantly advances the field of foley audio
dubbing, providing robust solutions for the challenges of multi-modal dubbing.
Even when utilizing the relatively lightweight GPT-2 model, our framework
outperforms open-source multimodal large models such as LLaVA, DeepSeek-VL, and
Moondream2. The dataset is available at https://github.com/borisfrb/MINT .
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined