Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in Transformers
arxiv(2023)
摘要
An extension of Transformers is proposed that enables explicit relational
reasoning through a novel module called the Abstractor. At the core of the
Abstractor is a variant of attention called relational cross-attention. The
approach is motivated by an architectural inductive bias for relational
learning that disentangles relational information from object-level features.
This enables explicit relational reasoning, supporting abstraction and
generalization from limited data. The Abstractor is first evaluated on simple
discriminative relational tasks and compared to existing relational
architectures. Next, the Abstractor is evaluated on purely relational
sequence-to-sequence tasks, where dramatic improvements are seen in sample
efficiency compared to standard Transformers. Finally, Abstractors are
evaluated on a collection of tasks based on mathematical problem solving, where
consistent improvements in performance and sample efficiency are observed.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要