Towards Theoretical Analysis of Transformation Complexity of ReLU DNNs

Abstract

This paper aims to theoretically analyze the complexity of feature transformations encoded in DNNs with ReLU layers. We propose metrics to measure three types of complexities of transformations based on the information theory. We further discover and prove the strong correlation between the complexity and the disentanglement of transformations. Based on the proposed metrics, we analyze two typical phenomena of the change of the transformation complexity during the training process, and explore the ceiling of a DNN's complexity. The proposed metrics can also be used as a loss to learn a DNN with the minimum complexity, which also controls the over-fitting level of the DNN and influences adversarial robustness, adversarial transferability, and knowledge consistency. Comprehensive comparative studies have provided new perspectives to understand the DNN.

Why Adversarial Training of ReLU Networks Is Difficult?

“…To this end, we learned four types of ReLU networks on the MNIST dataset via adversarial training. We followed settings in [19] to construct five MLPs, five CNNs, three MLPs with skip connections, and three CNNs with skip connections, respectively. Experimental results show that the average κ over all sixteen networks was 0.097, which verified the correctness of Theorem 3.…”

“…To this end, we learned four types of ReLU networks, including MLPs, CNNs, MLPs with skip connections (namely ResMLP), and CNNs with skip connections (namely ResCNN), on the MNIST dataset [11] via adversarial training. Here, we followed settings in [19] to construct five different MLPs, which consisted of 1, 2, 3, 4, 5 fully-connected (FC) layers, respectively. Each FC layer contained 200 neurons.…”