Data-Free Low-Bit Quantization via Dynamic Multi-teacher Knowledge Distillation

Data-free quantization is an effective way to compress deep neural networks under the situation where training data is unavailable, due to data privacy and security issues. Although Off-the-shelf data-free quantization methods achieve the relatively same accuracy as the fully-precision (FP) models for high-bit (e.g., 8-bit) quantization, low-bit quantization performance drops significantly to restrict their extensive applications. In this paper, we propose a novel data-free low-bit quantization method via Dynamic Multi-teacher Knowledge Distillation (DMKD) to improve the performance of low-bit quantization models. In particular, we first introduce a generator to synthesize the training data based on the input of random noise. The low-bit quantization models are then trained on these synthetic images by the dynamic knowledge from the FP model and the high-bit quantization models, which are balanced by learnable loss weight factors. The factors are controlled by a tiny learnable FP network to adaptively allocate the balanced weights for the knowledge from the FP model and the high-bit quantization models during training. For inference, we only kept the low-bit quantization model by safely removing other additional networks, such as the generator and the tiny model. Extensive experiments demonstrate the effectiveness of DMKD for low-bit quantization of widely-used convolutional neural networks (CNNs) on different benchmark datasets. Our DMKD ooon methods.