Just say zero: containing critical bit-error propagation in deep neural networks with anomalous feature suppression

ABSTRACTDNNs are abundantly employed in a variety of applications, including real-time systems with strict safety constraints. The consequences of errors prove disastrous in safety-critical systems, such as autonomous driving, healthcare, and industrial applications. DNNs are resilient to limited numerical perturbations yet fragile under large deviations in weights and activations. The traditional error tolerance measures fail to meet the tight design constraints of DNN processing systems due to extensive overheads or limited advantages in abundant error conditions. The algorithmic particularities of DNNs though create novel opportunities to deal with errors more effectively and economically. We revisit the two fundamental tasks in fault-tolerant system design, namely, error detection and correction, and demonstrate that the precise versions of these operations could be replaced by approximated counterparts in DNNs to deliver an extensive bit-error resilience even at high error rates while necessitating no information redundancy. We first maintain DNN accuracy even under extreme error rates by suppressing the numerical contributions of anomalous activations, eliminating any reliance on precise error correction. We tackle the problem of no redundancy error detection by establishing in training numerical associations among activations, and employing them for anomaly detection. Anomalous feature detection and suppression, performed efficiently at inference with minimal resources in a DNN accelerator, is shown to deliver significant resilience boosts while imposing neither information redundancy nor perceptible overheads.