DNN Diagnosis and Cure Based on Aggregated Concentration Ratio and Residual Connection
2022 7th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA)(2022)
Abstract
Designing deep neural network (DNN) models, empirically or automatically, has become a popular research area in recent years. Successful examples include ResNet, ConvNeXt, MnasNet, and NetAdaptV2. A standard pipeline of model users is to pretrain a designed DNN network, and concatenate it with additional layers which are specified for a downstream application. However, we find such paradigm, i.e., pretraining and concatenation, cannot always guarantee a satisfactory prediction ability, even if the pre-trained network has been extensively applied before. In this work, we propose a self diagnosis and cure method for empirical DNN design. Our diagnosis method can automatically identify the layer that incurs the largest accuracy degradation. The diagnosis criterion, aggregated concentration ratio, adopts the difficulty of linearly representing a layer as the indicator, instead of traditional means such as subjective evaluation and extra datasets. Moreover, the problematic layer will be cured by adding a residual connection with a 3x3 convolutional kernel. Experiments on object detection show that accuracy of our cured detector with 2.8M parameters is close to ResNet-101 with 44.5M parameters. Experiments on unsupervised domain adaptation further show that accuracy of our cured domain adaptor with 2.8M parameters is close to AlexNet with 61M parameters. Sensitivity analysis validates that curing our identified layer achieves better accuracy compared to curing other 2, 3, 4, and 5 layers in a DNN.
MoreTranslated text
Key words
Network architecture design,pretraining,DNN diagnosis,DNN cure,aggregated concentration ratio,residual connection
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined