Outlier-Efficient Hopfield Layers for Large Transformer-Based Models

We introduce an Outlier-Efficient Modern Hopfield Model (termed 𝙾𝚞𝚝𝙴𝚏𝚏𝙷𝚘𝚙) and use it to address the outlier-induced challenge of quantizing gigantic transformer-based models. Our main contribution is a novel associative memory model facilitating outlier-efficient associative memory retrievals. Interestingly, this memory model manifests a model-based interpretation of an outlier-efficient attention mechanism (Softmax_1): it is an approximation of the memory retrieval process of 𝙾𝚞𝚝𝙴𝚏𝚏𝙷𝚘𝚙. Methodologically, this allows us to debut novel outlier-efficient Hopfield layers a powerful attention alternative with superior post-quantization performance. Theoretically, the Outlier-Efficient Modern Hopfield Model retains and improves the desirable properties of the standard modern Hopfield models, including fixed point convergence and exponential storage capacity. Empirically, we demonstrate the proposed model's efficacy across large-scale transformer-based and Hopfield-based models (including BERT, OPT, ViT and STanHop-Net), benchmarking against state-of-the-art methods including 𝙲𝚕𝚒𝚙𝚙𝚎𝚍_𝚂𝚘𝚏𝚝𝚖𝚊𝚡 and 𝙶𝚊𝚝𝚎𝚍_𝙰𝚝𝚝𝚎𝚗𝚝𝚒𝚘𝚗. Notably, 𝙾𝚞𝚝𝙴𝚏𝚏𝙷𝚘𝚙 achieves on average ∼22+% reductions in both average kurtosis and maximum infinity norm of model outputs accross 4 models.