Generalization Error Bounds for Iterative Recovery Algorithms Unfolded as Neural Networks

Abstract

Motivated by the learned iterative soft thresholding algorithm (LISTA), we introduce a general class of neural networks suitable for sparse reconstruction from few linear measurements. By allowing a wide range of degrees of weight-sharing between the layers, we enable a unified analysis for very different neural network types, ranging from recurrent ones to networks more similar to standard feedforward neural networks. Based on training samples, via empirical risk minimization we aim at learning the optimal network parameters and thereby the optimal network that reconstructs signals from their low-dimensional linear measurements. We derive generalization bounds by analyzing the Rademacher complexity of hypothesis classes consisting of such deep networks, that also take into account the thresholding parameters. We obtain estimates of the sample complexity that essentially depend only linearly on the number of parameters and on the depth. We apply our main result to obtain specific generalization bounds for several practical examples, including different algorithms for (implicit) dictionary learning, and convolutional neural networks.

Theoretical Perspectives on Deep Learning Methods in Inverse Problems

“…This result, whose proof relies on a Rademacher complexity analysis, suggests that model-based networks may exhibit better generalization capabilities than traditional neural networks, in line with empirical results [45]. Concretely, this generalization error bound for LISTA-like model-based networks depends on the number of layers only logarithmically, whereas generalization error bounds for traditional neural networks (albeit in classification settings) can scale exponentially in the number of layers [13], [109]. On the other hand, the dependence on m and n in (51) remains fairly strong, and it would be of interest to determine if it can be reduced, e.g., by exploiting sparsity (notice that the sparsity level s is absent in (51)).…”

“…Learning-Theoretic Oriented Results: Another class of contributions concentrates on learning-theoretic aspects, studying how the generalization error -corresponding to the difference between the expected error and the empirical error -behaves as a function of various quantities relating to the learning problem, including the number of training samples. Works giving results of this kind include [13], [109], [23], [115].…”

“…Other works. Extensions of the above generalization results are covered in [13], [109], [115], involving different learnable parameters or different degrees of weight-sharing between different layers and a variety of network architectures. Notably, [115] offers a Rademacher complexity and local Rademacher complexity analysis of the generalization error and estimation error of model-based networks, respectively, showing that the soft-thresholding nonlinearity can play a key role in guaranteeing that model-based networks perform better than traditional neural networks.…”

Theoretical Perspectives on Deep Learning Methods in Inverse Problems

“…Learning-Theoretic Oriented Results: Another class of contributions concentrates on learning-theoretic aspects, studying how the generalization error -corresponding to the difference between the expected error and the empirical error -behaves as a function of various quantities relating to the learning problem, including the number of training samples. Works giving results of this kind include [14], [117], [26], [124].…”

DECONET: an Unfolding Network for Analysis-based Compressed Sensing with Generalization Error Estimates

“…For example, [44,45] provide convergence guarantees regarding variants of learned ISTA networks, while [26] studies the robustness of an unfolding network, produced by a forward-backward proximal interior point method. Moreover, [23,46,47] present generalization 1 error bounds, in terms of the Rademacher complexity of the hypothesis class consisting of the functions that a learnable ISTA network can implement. The concept of generalization error bounds is widely known in the field of statistical learning theory and studied by different -albeit connected -complexity terms, such as Rademacher complexity [48], Vapnik-Chervonenkis (VC) dimension [49], stability [50] and robustness [51].…”