Interpreting convolutional neural networks by explaining their predictions

We propose a method that exploits the feedback provided by visual explanation methods combined with pattern mining techniques to identify the relevant class-specific and class-shared internal units. In addition, we put forward a patch extraction approach to find faithfully class-specific and class-shared visual patterns. Contrary to the common practice in literature, our approach does not require pushing augmented visual patches through the model. Experiments on two CNN architectures show the effectiveness of the proposed method.