DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback
CVPR 2024(2023)
摘要
We present DRESS, a large vision language model (LVLM) that innovatively
exploits Natural Language feedback (NLF) from Large Language Models to enhance
its alignment and interactions by addressing two key limitations in the
state-of-the-art LVLMs. First, prior LVLMs generally rely only on the
instruction finetuning stage to enhance alignment with human preferences.
Without incorporating extra feedback, they are still prone to generate
unhelpful, hallucinated, or harmful responses. Second, while the visual
instruction tuning data is generally structured in a multi-turn dialogue
format, the connections and dependencies among consecutive conversational turns
are weak. This reduces the capacity for effective multi-turn interactions. To
tackle these, we propose a novel categorization of the NLF into two key types:
critique and refinement. The critique NLF identifies the strengths and
weaknesses of the responses and is used to align the LVLMs with human
preferences. The refinement NLF offers concrete suggestions for improvement and
is adopted to improve the interaction ability of the LVLMs– which focuses on
LVLMs' ability to refine responses by incorporating feedback in multi-turn
interactions. To address the non-differentiable nature of NLF, we generalize
conditional reinforcement learning for training. Our experimental results
demonstrate that DRESS can generate more helpful (9.76
harmless (21.03
multi-turn interactions compared to SOTA LVMLs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要