LieRE: Generalizing Rotary Position Encodings
CoRR(2024)
摘要
While Rotary Position Embeddings (RoPE) for natural language performs well
and has become widely adopted, its adoption for other modalities has been
slower. Here, we introduce Lie group Relative position Encodings (LieRE) that
goes beyond RoPE in supporting higher dimensional inputs. We evaluate the
performance of LieRE on 2D and 3D image classification tasks and observe that
LieRE leads to marked improvements in performance (up to 6
efficiency (3.5x reduction), data efficiency (30
RoFormer, DeiT III, RoPE-Mixed and Vision-Llama
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要