Topology of deep neural networks

Naitzat, Gregory; Zhitnikov, Andrey; Lim, Lek-Heng

doi:10.48550/arxiv.2004.06093

Cited by 7 publications

(9 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar to a recent work that exploits graph-based manifolds for the topology analysis of DNNs (Naitzat et al, 2020) (Zhang et al, 2013).…”

Section: Graph-based Manifold Constructionmentioning

confidence: 95%

“…Pair-wise distance calculations on the manifold will be key to estimating γ F max . To this end, the geodesic distance metric is arguably the most natural choice: for graph-based manifold problems the shortest-path distance metric have been adopted for approximating the geodesic distance on a manifold in nonlinear dimensionality reduction and neural network topological analysis (Tenenbaum et al, 2000;Naitzat et al, 2020). However, to exhaustively search for γ F max will require computing all-pairs shortest-paths between N input (output) data points, which can be prohibitively expensive even when taking advantage of the state-of-the-art randomized method (Williams, 2018).…”

Section: Computing γ Fmentioning

confidence: 99%

See 1 more Smart Citation

SPADE: A Spectral Method for Black-Box Adversarial Robustness Evaluation

Cheng,

Deng,

Zhao

et al. 2021

Preprint

View full text Add to dashboard Cite

A black-box spectral method is introduced for evaluating the adversarial robustness of a given machine learning (ML) model. Our approach, named SPADE, exploits bijective distance mapping between the input/output graphs constructed for approximating the manifolds corresponding to the input/output data. By leveraging the generalized Courant-Fischer theorem, we propose a SPADE score for evaluating the adversarial robustness of a given model, which is proved to be an upper bound of the best Lipschitz constant under the manifold setting. To reveal the most nonrobust data samples highly vulnerable to adversarial attacks, we develop a spectral graph embedding procedure leveraging dominant generalized eigenvectors. This embedding step allows assigning each data sample a robustness score that can be further harnessed for more effective adversarial training. Our experiments show the proposed SPADE method leads to promising empirical results for neural network models adversarially trained with the MNIST and CIFAR-10 data sets.

show abstract

“…Similar to a recent work that exploits graph-based manifolds for the topology analysis of DNNs (Naitzat et al, 2020) (Zhang et al, 2013).…”

Section: Graph-based Manifold Constructionmentioning

confidence: 95%

Section: Computing γ Fmentioning

confidence: 99%

SPADE: A Spectral Method for Black-Box Adversarial Robustness Evaluation

Cheng,

Deng,

Zhao

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Olah performed a number of topological experiments illustrating the importance of considering the topology of the underlying data when making a neural network. In [26] the activations of a binary classification neural network were considered as point clouds that the layer functions of the network are acting on. The topologies of these activations are then studied using homological tools such as persistent homology [10].…”

Section: Previous Workmentioning

confidence: 99%

Topological Deep Learning: Classification Neural Networks

Hajij,

Istvan

2021

Preprint

View full text Add to dashboard Cite

Topological deep learning is a formalism that is aimed at introducing topological language to deep learning for the purpose of utilizing the minimal mathematical structures to formalize problems that arise in a generic deep learning problem. This is the first of a sequence of articles with the purpose of introducing and studying this formalism. In this article, we define and study the classification problem in machine learning in a topological setting. Using this topological framework, we show when the classification problem is possible or not possible in the context of neural networks. Finally, we show that for a given data, the architecture of a classification neural network must take into account the topology of this data in order to achieve a successful classification task.

show abstract

“…The combined deformation needed to curve the plane and to "make" the holes causes the generative function to be highly nonlinear. Note that when considering a DGM that uses a DNN with ReLU activation functions as generator g(z), it is also possible for g(z) to change topology of the input by "folding" transformations (Naitzat et al, 2020).…”

Section: Deep Generative Models (Dgm) To Represent Realistic Patternsmentioning

confidence: 99%

“…In order to approximate manifolds of realistic patterns, most common DGMs involve (artificial) neural networks with several layers and nonlinear (activation) functions. Recently, it has been shown that deep neural networks with a ReLU activation function are able to change topology of the input (Naitzat et al, 2020) so, besides nonlinearity, the generator of DGMs may also induce changes in topology when mapping from the latent space. When the sole purpose of the DGM is for generating new samples, high nonlinearity and induced changes in topology are not important but they might be an issue when the DGM is used for additional tasks, such as inversion.…”

Section: Introductionmentioning

confidence: 99%

Deep generative models in inversion: a review and development of a new approach based on a variational autoencoder

Lopez-Alvis,

Laloy,

Nguyen

et al. 2020

Preprint

View full text Add to dashboard Cite

When solving inverse problems in geophysical imaging, deep generative models (DGMs) may be used to enforce the solution to display highly structured spatial patterns which are supported by independent information (e.g. the geological setting) of the subsurface. In such case, inversion may be formulated in a latent space where a low-dimensional parameterization of the patterns is defined and where Markov chain Monte Carlo or gradient-based methods may be applied. However, the generative mapping between the latent and the original (pixel) representations is usually highly nonlinear which may cause some difficulties for inversion, especially for gradient-based methods. In this contribution we review the conceptual framework of inversion with DGMs and study the principal causes of the nonlinearity of the generative mapping. As a result, we identify a conflict between two goals: the accuracy of the generated patterns and the feasibility of gradient-based inversion. In addition, we show how some of the training parameters of a variational autoencoder, which is a particular instance of a DGM, may be chosen so that a tradeoff between these two goals is achieved and acceptable inversion results are obtained with a stochastic gradient-descent scheme.A test case using truth models with channel patterns of different complexity and cross-borehole traveltime tomographic data involving both a linear and a nonlinear forward operator is used to assess the performance of the proposed approach.

show abstract

Topology of deep neural networks

Cited by 7 publications

References 0 publications

SPADE: A Spectral Method for Black-Box Adversarial Robustness Evaluation

SPADE: A Spectral Method for Black-Box Adversarial Robustness Evaluation

Topological Deep Learning: Classification Neural Networks

Deep generative models in inversion: a review and development of a new approach based on a variational autoencoder

Contact Info

Product

Resources

About