Where We Have Arrived in Proving the Emergence of Sparse Interaction Primitives in AI Models

This study aims to prove the emergence of symbolic concepts (or more precisely, sparse primitive inference patterns) in well-trained AI models. Specifically, we prove the following three conditions for the emergence. (i) The high-order derivatives of the model output with respect to the input variables are all zero. (ii) The model can be used on occluded samples, and when the input sample is less occluded, the model will yield higher confidence. (iii) The confidence of the model does not significantly degrade on occluded samples. These conditions are quite common, and we prove that under these conditions, the model will only encode a relatively small number of sparse interactions between input variables. Moreover, we can consider such interactions as symbolic primitive inference patterns encoded by an AI model, because we show that inference scores of the model on an exponentially large number of randomly masked samples can always be well mimicked by numerical effects of just a few interactions.