DNN Diagnosis and Cure Based on Aggregated Concentration Ratio and Residual Connection

Designing deep neural network (DNN) models, empirically or automatically, has become a popular research area in recent years. Successful examples include ResNet, ConvNeXt, MnasNet, and NetAdaptV2. A standard pipeline of model users is to pretrain a designed DNN network, and concatenate it with additional layers which are specified for a downstream application. However, we find such paradigm, i.e., pretraining and concatenation, cannot always guarantee a satisfactory prediction ability, even if the pre-trained network has been extensively applied before. In this work, we propose a self diagnosis and cure method for empirical DNN design. Our diagnosis method can automatically identify the layer that incurs the largest accuracy degradation. The diagnosis criterion, aggregated concentration ratio, adopts the difficulty of linearly representing a layer as the indicator, instead of traditional means such as subjective evaluation and extra datasets. Moreover, the problematic layer will be cured by adding a residual connection with a 3x3 convolutional kernel. Experiments on object detection show that accuracy of our cured detector with 2.8M parameters is close to ResNet-101 with 44.5M parameters. Experiments on unsupervised domain adaptation further show that accuracy of our cured domain adaptor with 2.8M parameters is close to AlexNet with 61M parameters. Sensitivity analysis validates that curing our identified layer achieves better accuracy compared to curing other 2, 3, 4, and 5 layers in a DNN.