A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene
CoRR(2024)
Abstract
Pre-trained vision-language (V-L) models such as CLIP have shown excellent
performance in many downstream cross-modal tasks. However, most of them are
only applicable to the English context. Subsequent research has focused on this
problem and proposed improved models, such as CN-CLIP and AltCLIP, to
facilitate their applicability to Chinese and even other languages.
Nevertheless, these models suffer from high latency and a large memory
footprint in inference, which limits their further deployment on
resource-constrained edge devices. In this work, we propose a conceptually
simple yet effective multilingual CLIP Compression framework and train a
lightweight multilingual vision-language model, called DC-CLIP, for both
Chinese and English context. In this framework, we collect high-quality Chinese
and English text-image pairs and design two training stages, including
multilingual vision-language feature distillation and alignment. During the
first stage, lightweight image/text student models are designed to learn robust
visual/multilingual textual feature representation ability from corresponding
teacher models, respectively. Subsequently, the multilingual vision-language
alignment stage enables effective alignment of visual and multilingual textual
features to further improve the model's multilingual performance. Comprehensive
experiments in zero-shot image classification, conducted based on the ELEVATER
benchmark, showcase that DC-CLIP achieves superior performance in the English
context and competitive performance in the Chinese context, even with less
training data, when compared to existing models of similar parameter magnitude.
The evaluation demonstrates the effectiveness of our designed training
mechanism.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined