Zero-Overhead Protection for CNN Weights

Abstract

The numerical format used for representing weights and activations plays a key role in the computational efficiency and robustness of CNNs. Recently, a 16-bit floating point format called Brain-Float 16 (bf16) has been proposed and implemented in hardware accelerators. However, the robustness of accelerators implemented with this format has not yet been studied. In this paper, we perform a comparison of the robustness of state-of-the art CNNs implemented with 8-bit integer, Brain-Float 16 and 32bit floating point formats. We also introduce an error detection and masking technique, called opportunistic parity (OP), which can detect and mask errors in the weights with zero storage overhead. With this technique, the robustness of floating point weights to bit-flips can be improved by up to three orders of magnitude.

Testability and Dependability of AI Hardware: Survey, Trends, Challenges, and Perspectives

“…[245]-[249] Redundancy-based [77], [85], [91], [101], [104], [250]- [256] Fault masking [91], [92], [257], [258], [259], [260], [261] Variation-aware mapping for memristor crossbar arrays [262], [263] ECC [81], [83], [84], [264] ML-based [265] Adaptive training after testing [146], [266] Razor [257], [267], [268] Neuron adaptation [269] Aging-aware on-line training of memristor crossbar arrays [270],…”

“…The fault mitigation approach is to mask the faulty PE's output to zero. As a PE roughly corresponds to a In [260], the Opportunistic Parity (OP) fault mitigation technique is proposed for protecting CNN weights. OP is based on the observation that errors in the LSBs of the weights can be tolerated.…”

Review on Application of Deep Neural Networks and Parallel Architectures for Rock Fragmentation Problems

“…Операции со смешанной точностью предполагают, что хотя бы один из операндов имеет пониженную точность. Примерами таких форматов являются TensorFloat-32 (TF32) [124], Brain Float (BF32 и BF16) [125] и другие.…”