TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild
CoRR(2023)
Abstract
Large language models with instruction-following abilities have
revolutionized the field of artificial intelligence. These models show
exceptional generalizability to tackle various real-world tasks through their
natural language interfaces. However, their performance heavily relies on
high-quality exemplar data, which is often difficult to obtain. This challenge
is further exacerbated when it comes to multimodal instruction following. We
introduce TextBind, an almost annotation-free framework for empowering larger
language models with the multi-turn interleaved multimodal
instruction-following capabilities. Our approach requires only image-caption
pairs and generates multi-turn multimodal instruction-response conversations
from a language model. To accommodate interleaved image-text inputs and
outputs, we devise MIM, a language model-centric architecture that seamlessly
integrates image encoder and decoder models. We release our dataset, model, and
demo to foster future research in the area of multimodal instruction following.
MoreTranslated text
Key words
multi-turn,instruction-following
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined