Error-Correcting Output Codes with Ensemble Diversity for Robust Learning in Neural Networks

Abstract

Though deep learning has been applied successfully in many scenarios, malicious inputs with human-imperceptible perturbations can make it vulnerable in real applications. This paper proposes an error-correcting neural network (ECNN) that combines a set of binary classifiers to combat adversarial examples in the multi-class classification problem. To build an ECNN, we propose to design a code matrix so that the minimum Hamming distance between any two rows (i.e., two codewords) and the minimum shared information distance between any two columns (i.e., two partitions of class labels) are simultaneously maximized. Maximizing row distances can increase the system fault tolerance while maximizing column distances helps increase the diversity between binary classifiers. We propose an end-to-end training method for our ECNN, which allows further improvement of the diversity between binary classifiers. The end-to-end training renders our proposed ECNN different from the traditional error-correcting output code (ECOC) based methods that train binary classifiers independently. ECNN is complementary to other existing defense approaches such as adversarial training and can be applied in conjunction with them. We empirically demonstrate that our proposed ECNN is effective against the state-of-the-art white-box and black-box attacks on several datasets while maintaining good classification accuracy on normal examples.

Learning Label Encodings for Deep Regression

“…Encoding design is a well-studied problem with applications in several fields. Iterative approaches, such as simulated annealing or random walk, have been proposed for code design (Dietterich & Bakiri, 1995;Song et al, 2021). However, iterative approaches are computationally expensive as each iteration requires full/partial training of the network to measure the error for sample encodings.…”

“…We evaluate different label encoding design approaches, including simulated annealing and autoencoder. These approaches have been used to design encodings for multiclass classification by prior works (Song et al, 2021;Cissé et al, 2012). We adapt these approaches to design encodings for regression tasks and compare RLEL with these code design techniques.…”

“…The use of binary-encoded labels for regression has multiple advantages. Additionally, predicting a set of values (e.g., classifiers' output) instead of one value (direct regression) introduces ensemble diversity, which improves accuracy (Song et al, 2021). Furthermore, encoded labels introduce redundancy in the label presentation, which improves error correcting capability and accuracy (Dietterich & Bakiri, 1995).…”

“…Here blue and white colors represent 1 and 0, respectively. search, and simulated annealing, have been proposed to design suitable encoding for multiclass classification (Cissé et al, 2012;Dietterich & Bakiri, 1995;Song et al, 2021). However, these encodings perform relatively poorly for regression due to differences in task objectives (Section 2).…”

Improved Robustness Against Adaptive Attacks With Ensembles and Error-Correcting Output Codes

“…Other ECOC architectures proposed for adversarial robustness share some parts of the binary classifiers (i.e. feature extractor layers) as a matter of efficiency (Verma & Swami, 2019;Song et al, 2021). We believe that using independent CNN binary classifiers to construct the ECOC ensembles will increase robustness by promoting the learning of more diverse features, making the binary classifiers more challenging to fool simultaneously with adversarial attacks.…”