Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance
arxiv(2023)
摘要
We present a novel framework to overcome the limitations of equivariant
architectures in learning functions with group symmetries. In contrary to
equivariant architectures, we use an arbitrary base model such as an MLP or a
transformer and symmetrize it to be equivariant to the given group by employing
a small equivariant network that parameterizes the probabilistic distribution
underlying the symmetrization. The distribution is end-to-end trained with the
base model which can maximize performance while reducing sample complexity of
symmetrization. We show that this approach ensures not only equivariance to
given group but also universal approximation capability in expectation. We
implement our method on various base models, including patch-based transformers
that can be initialized from pretrained vision transformers, and test them for
a wide range of symmetry groups including permutation and Euclidean groups and
their combinations. Empirical tests show competitive results against tailored
equivariant architectures, suggesting the potential for learning equivariant
functions for diverse groups using a non-equivariant universal base
architecture. We further show evidence of enhanced learning in symmetric
modalities, like graphs, when pretrained from non-symmetric modalities, like
vision. Code is available at https://github.com/jw9730/lps.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要