ViLA: Efficient Video-Language Alignment for Video Question Answering
arxiv(2023)
Abstract
In this work, we propose an efficient Video-Language Alignment (ViLA)
network. Our ViLA model addresses both efficient frame sampling and effective
cross-modal alignment in a unified way. In our ViLA network, we design a new
learnable text-guided Frame-Prompter together with a new cross-modal
distillation (QFormer-Distiller) module. Pre-trained large image-language
models have shown promising results on problems such as visual question
answering (VQA). However, how to efficiently and effectively sample video
frames when adapting pre-trained large image-language model to video-language
alignment is still the major challenge. Compared with prior work, our ViLA
model demonstrates the capability of selecting key frames with critical
contents, thus improving the video-language alignment accuracy while reducing
the inference latency +3.3
our ViLA network outperforms the state-of-the-art methods on the video
question-answering benchmarks: +4.6
with 3.0X speed up, ours 2-frames out-perform SeViLA 4-frames on the VLEP
dataset with 4.2X speed-up.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined