Ovis: Structural Embedding Alignment for Multimodal Large Language Model
CoRR(2024)
Abstract
Current Multimodal Large Language Models (MLLMs) typically integrate a
pre-trained LLM with another pre-trained vision transformer through a
connector, such as an MLP, endowing the LLM with visual capabilities. However,
the misalignment between two embedding strategies in MLLMs – the structural
textual embeddings based on an embedding look-up table and the continuous
embeddings generated directly by the vision encoder – makes challenges for a
more seamless fusion of visual and textual information. We propose Ovis, a
novel MLLM architecture designed to structurally align visual and textual
embeddings. Ovis integrates an additional learnable visual embedding table into
the visual encoder's process. To capture rich visual semantics, each image
patch indexes the visual embedding table multiple times, resulting in a final
visual embedding that is a probabilistic combination of the indexed embeddings.
This structural approach mirrors the method used for generating textual
embeddings. Empirical evaluations on various multimodal benchmarks demonstrate
that Ovis outperforms open-source MLLMs of similar parameter scales and even
surpasses the proprietary model Qwen-VL-Plus overall. These results highlight
the potential of Ovis' structured visual representation for advancing MLLM
architectural design and promoting more effective multimodal learning. Both the
source code and the training dataset of Ovis will be made publicly available.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined