Homomorphic Evaluation Friendly Vision Transformer Design.

The widespread adoption of neural networks has yielded remarkable success across diverse tasks. However, the increasing security concerns associated with state-of-the-art model deployment underscore the need for Privacy-Preserving Machine Learning (PPML). Homomorphic Encryption (HE) is one of the popular security primitives that empower PPML. Such primitives allow the secure computation over ciphertext without decryption and send the result back to the data owner, protecting the privacy of user data. In this work, we explore the possible solutions to apply the trending vision transformer in the homomorphic domain by modifying the nonlinear activation functions to make them compatible with homomorphic-friendly operations.