CLIP: Cheap Lipschitz Training of Neural Networks

Bungert, Leon; Raab, René; Tim, Roith,; Schwinn, Leo; Tenbrinck, Daniel

doi:10.1007/978-3-030-75549-2_25

Cited by 15 publications

(7 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The robustness of neural networks to distribution shifts is a broad research area (Hendrycks et al, 2021a). Among the sub-fields of this research area are domain adaptation (Shimodaira, 2000;Wilson & Cook, 2020), out-ofdistribution (OOD) detection (Hendrycks & Gimpel, 2017;Schwinn et al, 2021a), corruption and perturbation robustness (Hendrycks & Dietterich, 2019;Yin et al, 2019;Geirhos et al, 2019;Zhang et al, 2021), robustness to spatial transformations (Engstrom et al, 2019), adversarial robustness (Szegedy et al, 2014;Goodfellow et al, 2015;Madry et al, 2018;Bungert et al, 2021), and more. Here, we focus on robustness against common corruptions and perturbations, spatial transformations, natural adversarial examples, and optimized adversarial examples.…”

Section: Related Workmentioning

confidence: 99%

Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification

Schwinn¹,

Bungert²,

Nguyen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

The reliability of neural networks is essential for their use in safety-critical applications. Existing approaches generally aim at improving the robustness of neural networks to either real-world distribution shifts (e.g., common corruptions and perturbations, spatial transformations, and natural adversarial examples) or worst-case distribution shifts (e.g., optimized adversarial examples). In this work, we propose the Decision Region Quantification (DRQ) algorithm to improve the robustness of any differentiable pre-trained model against both real-world and worst-case distribution shifts in the data. DRQ analyzes the robustness of local decision regions in the vicinity of a given data point to make more reliable predictions. We theoretically motivate the DRQ algorithm by showing that it effectively smooths spurious local extrema in the decision surface. Furthermore, we propose an implementation using targeted and untargeted adversarial attacks. An extensive empirical evaluation shows that DRQ increases the robustness of adversarially and nonadversarially trained models against real-world and worst-case distribution shifts on several computer vision benchmark datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification

Schwinn¹,

Bungert²,

Nguyen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This strategy constitutes an instance of the popular weight normalisation technique [ 95 ], and related spectral normalisations have shown to be successful for improving the performance and convergence speed of the training process [ 15 , 22 , 42 , 113 ].…”

Section: From Diffusion To Symmetric Residual Networkmentioning

confidence: 99%

Connections Between Numerical Algorithms for PDEs and Neural Networks

et al. 2022

View full text Add to dashboard Cite

We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights, we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

show abstract

“…Consequently, the eigenvalues of K G(u k )K lie in the interval 0, τ L||K|| 2 2 . Then the operator I −τ K G(u k )K has eigenvalues in 1 − τ L||K|| 2 2 , 1 , and the condition 1 − τ L||K|| 2 2 ≥ −1 (18) leads to the bound (14).…”

Section: Stability For Symmetric Resnetsmentioning

confidence: 99%

“…This strategy constitutes an instance of the popular weight normalisation technique [91], and related spectral normalisations have shown to be successful for improving the performance and convergence speed of the training process [14,39].…”

Section: Enforcing Stability In Practicementioning

confidence: 99%

Connections between Numerical Algorithms for PDEs and Neural Networks

Alt¹,

Schrader²,

Augustin³

et al. 2021

Preprint

View full text Add to dashboard Cite

We investigate numerous structural connections between numerical algorithms for partial differential equations (PDEs) and neural architectures. Our goal is to transfer the rich set of mathematical foundations from the world of PDEs to neural networks. Besides structural insights we provide concrete examples and experimental evaluations of the resulting architectures. Using the example of generalised nonlinear diffusion in 1D, we consider explicit schemes, acceleration strategies thereof, implicit schemes, and multigrid approaches. We connect these concepts to residual networks, recurrent neural networks, and U-net architectures. Our findings inspire a symmetric residual network design with provable stability guarantees and justify the effectiveness of skip connections in neural networks from a numerical perspective. Moreover, we present U-net architectures that implement multigrid techniques for learning efficient solutions of partial differential equation models, and motivate uncommon design choices such as trainable nonmonotone activation functions. Experimental evaluations show that the proposed architectures save half of the trainable parameters and can thus outperform standard ones with the same model complexity. Our considerations serve as a basis for explaining the success of popular neural architectures and provide a blueprint for developing new mathematically well-founded neural building blocks.

show abstract

CLIP: Cheap Lipschitz Training of Neural Networks

Cited by 15 publications

References 13 publications

Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification

Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification

Connections Between Numerical Algorithms for PDEs and Neural Networks

Connections between Numerical Algorithms for PDEs and Neural Networks

Contact Info

Product

Resources

About