Analytical bounds on the local Lipschitz constants of ReLU networks

Avant, Trevor; Morgansen, Kristi A.

doi:10.48550/arxiv.2104.14672

Cited by 2 publications

(5 citation statements)

References 5 publications

(7 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the bound is not strict since for the first architecture the bound is 34, 80 and the quotient lies between 1 and Hence, the constant is a worst-case scenario of the dilation caused by the function. Notice that this is consistent with the (strict) bounds of [AM21], that are of order 10 9 . Furthermore, there seems to be little to no difference in distance dilation among the four different architectures, see table 2, even though the constant of the algorithm does change.…”

Section: Boston Datasetsupporting

confidence: 86%

“…The results in [AM21] suggest that learners can be seen as Lipschitz function from one (pseudo)metric space to another. Thus, the answer to the first question of this section seems to be in the very definition of the Lipschitz function: the constant.…”

Section: Enriched Datasetsmentioning

confidence: 94%

“…Another way of phrasing it is: that the Lipschitz constant is a bound on how much we allow the algorithm to strengthen the differences among points in the dataset. This shows how the framework incorporates known metrics, as shown in [vLB04] and [AM21], into the categorical setting. Thus, we are enlarging the describable phenomena with this unifying framework.…”

Section: Theorem 36 a Neural Network (Viewed As A Parameterized Funct...mentioning

confidence: 98%

“…The third fact implies that the most common activation functions of a neural network are Lipschitz: sigmoid, Relu, Leaky-Relu, and hyperbolic tangent are all Lipschitz. For a more detailed work on the bounds of the ReLu function see [AM21]. More relevant to us, the softmax function is Lipschitz as a map of Euclidean spaces as seen in [GP17, proposition 4].…”

Section: Enriched Datasetsmentioning

confidence: 99%

See 3 more Smart Citations

Exploring explainable AI: category theory insights into machine learning algorithms

Fabregat-Hernández,

Palanca,

Botti

2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

Explainable artificial intelligence (XAI) is a growing field that aims to increase the transparency and interpretability of machine learning (ML) models. The aim of this work is to use the categorical properties of learning algorithms in conjunction with the categorical perspective of the information in the datasets to give a framework for explainability. In order to achieve this, we are going to define the enriched categories, with decorated morphisms, L e a r n L e a r n , P a r a P a r a and M N e t M N e t of learners, parameterized functions, and neural networks over metric spaces respectively. The main idea is to encode information from the dataset via categorical methods, see how it propagates, and lastly, interpret the results thanks again to categorical (metric) information. This means that we can attach numerical (computable) information via enrichment to the structural information of the category. With this, we can translate theoretical information into parameters that are easily understandable. We will make use of different categories of enrichment to keep track of different kinds of information. That is, to see how differences in attributes of the data are modified by the algorithm to result in differences in the output to achieve better separation. In that way, the categorical framework gives us an algorithm to interpret what the learning algorithm is doing. Furthermore, since it is designed with generality in mind, it should be applicable in various different contexts. There are three main properties of category theory that help with the interpretability of ML models: formality, the existence of universal properties, and compositionality. The last property offers a way to combine smaller, simpler models that are easily understood to build larger ones. This is achieved by formally representing the structure of ML algorithms and information contained in the model. Finally, universal properties are a cornerstone of category theory. They help us characterize an object, not by its attributes, but by how it interacts with other objects. Thus, we can formally characterize an algorithm by how it interacts with the data. The main advantage of the framework is that it can unify under the same language different techniques used in XAI. Thus, using the same language and concepts we can describe a myriad of techniques and properties of ML algorithms, streamlining their explanation and making them easier to generalize and extrapolate.

show abstract

Section: Boston Datasetsupporting

confidence: 86%

Section: Enriched Datasetsmentioning

confidence: 94%

Section: Theorem 36 a Neural Network (Viewed As A Parameterized Funct...mentioning

confidence: 98%

Section: Enriched Datasetsmentioning

confidence: 99%

See 2 more Smart Citations

Exploring explainable AI: category theory insights into machine learning algorithms

Fabregat-Hernández,

Palanca,

Botti

2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

show abstract

“…However, again this approach does not scale well with the number of layers. While these works estimate global Lipschitz constants, it was shown in [20] that estimating local Lipschitz can be done more efficiently.…”

Section: A Related Workmentioning

confidence: 99%

Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks

Xue¹,

Lindemann²,

Robey³

et al. 2022

Preprint

View full text Add to dashboard Cite

Lipschitz constants of neural networks allow for guarantees of robustness in image classification, safety in controller design, and generalizability beyond the training data. As calculating Lipschitz constants is NP-hard, techniques for estimating Lipschitz constants must navigate the trade-off between scalability and accuracy. In this work, we significantly push the scalability frontier of a semidefinite programming technique known as LipSDP while achieving zero accuracy loss. We first show that LipSDP has chordal sparsity, which allows us to derive a chordally sparse formulation that we call Chordal-LipSDP. The key benefit is that the main computational bottleneck of LipSDP, a large semidefinite constraint, is now decomposed into an equivalent collection of smaller onesallowing Chordal-LipSDP to outperform LipSDP particularly as the network depth grows. Moreover, our formulation uses a tunable sparsity parameter that enables one to gain tighter estimates without incurring a significant computational cost. We illustrate the scalability of our approach through extensive numerical experiments.

show abstract

Analytical bounds on the local Lipschitz constants of ReLU networks

Cited by 2 publications

References 5 publications

Exploring explainable AI: category theory insights into machine learning algorithms

Exploring explainable AI: category theory insights into machine learning algorithms

Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks

Contact Info

Product

Resources

About