Channel Redundancy and Overlap in Convolutional Neural Networks with Channel-Wise NNK Graphs

Bonet, David; Ortega, Antonio; Ruiz-Hidalgo, Javier; Shekkizhar, Sarath

doi:10.1109/icassp43922.2022.9746186

Cited by 7 publications

(7 citation statements)

References 23 publications

(33 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The local intrinsic dimension of a manifold can be estimated as the number of neighbors selected by NNK. It was shown in [43,50], that the number of neighbors per polytope, i.e., n i , correlates with the local dimension of the manifold around a data point i. This observation is consistent with geometric intuitions; the number of NNK neighbors is decided based on the availability of data that span orthogonal directions, i.e., the local subspace of manifold:…”

Section: Intrinsic Dimension Metricsupporting

confidence: 68%

The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning

Cosentino¹,

Shekkizhar²,

Soltanolkotabi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Self-supervised learning (SSL) has emerged as a desirable paradigm in computer vision due to the inability of supervised models to learn representations that can generalize in domains with limited labels. The recent popularity of SSL has led to the development of several models that make use of diverse training strategies, architectures, and data augmentation policies with no existing unified framework to study or assess their effectiveness in transfer learning. We propose a data-driven geometric strategy to analyze different SSL models using local neighborhoods in the feature space induced by each. Unlike existing approaches that consider mathematical approximations of the parameters, individual components, or optimization landscape, our work aims to explore the geometric properties of the representation manifolds learned by SSL models. Our proposed manifold graph metrics (MGMs) provide insights into the geometric similarities and differences between available SSL models, their invariances with respect to specific augmentations, and their performances on transfer learning tasks. Our key findings are two fold: (i) contrary to popular belief, the geometry of SSL models is not tied to its training paradigm (contrastive, non-contrastive, and cluster-based); (ii) we can predict the transfer learning capability for a specific model based on the geometric properties of its semantic and augmentation manifolds.

show abstract

Section: Intrinsic Dimension Metricsupporting

confidence: 68%

The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning

Cosentino¹,

Shekkizhar²,

Soltanolkotabi³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Garg et al [42] also leveraged unlabeled data for early stopping but focused on providing a theoretical perspective on generalization. Bonet et al [43] proposed a channel-wise early stopping method, which utilizes hidden features in a convolutional neural network (CNN); this method is focused on a CNN model in computer vision. Mahsereci et al [25] proposed an evidence-based (EB) stop criterion that utilizes the gradients of the training samples.…”

Section: Related Workmentioning

confidence: 99%

Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

Choi

Lee

2023

IEEE Access

View full text Add to dashboard Cite

To improve deep-learning performance in low-resource settings, many researchers have redesigned model architectures or applied additional data (e.g., external resources, unlabeled samples). However, there have been relatively few discussions on how to make good use of small amounts of labeled samples, although it is potentially beneficial and should be done before applying additional data or redesigning models. In this study, we assume a low-resource setting in which only a few labeled samples (i.e., 30-100 per class) are available, and we discuss how to exploit them without additional data or model redesigns. We explore possible approaches in the following three aspects: training validation splitting, early stopping, and weight initialization. Extensive experiments are conducted on six public sentence classification datasets. Performance on various evaluation metrics (e.g., accuracy, loss, and calibration error) significantly varied depending on the approaches that were combined in the three aspects. Based on the results, we propose an integrated method, which is to initialize the model with a weight averaging method and use a nonvalidation stop method to train all samples. This simple integrated method consistently outperforms the competitive methods; e.g., the average accuracy of six datasets of this method was 1.8% higher than those of conventional validation-based methods. In addition, the integrated method further improves the performance when adapted to several state-of-the-art models that use additional data or redesign the network architecture (e.g., self-training and enhanced structural models). Our results highlight the importance of the training strategy and suggest that the integrated method can be the first step in the low-resource setting. This study provides empirical knowledge that will be helpful when dealing with low-resource data in future efforts. Our code is publicly available at https://github.com/DMCB-GIST/exploit_all_samples.

show abstract

“…NNK has been shown to perform well in several machine learning tasks [15], image representation [16], and generalization estimation in neural networks [17]. Furthermore, NNK has also been used to understand convolutional neural networks (CNN) channel redundancy [18] and to propose an early stopping criterion for them [19]. Graph properties (not necessarily based on NNK graphs) have been also proposed for the understanding and interpretation of deep neural network performance [20], latent space geometry [21,22] and to improve model robustness [23].…”

Section: Non-negative Kernel (Nnk) Regression Graphsmentioning

confidence: 99%

Study of Manifold Geometry Using Multiscale Non-Negative Kernel Graphs

Hurtado

Shekkizhar

Ruiz-Hidalgo

et al. 2023

ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Modern machine learning systems are increasingly trained on large amounts of data embedded in high-dimensional spaces. Often this is done without analyzing the structure of the dataset. In this work, we propose a framework to study the geometric structure of the data. We make use of our recently introduced non-negative kernel (NNK) regression graphs to estimate the point density, intrinsic dimension, and linearity of the data manifold (curvature). We further generalize the graph construction and geometric estimation to multiple scales by iteratively merging neighborhoods in the input data. Our experiments demonstrate the effectiveness of our proposed approach over other baselines in estimating the local geometry of the data manifolds on synthetic and real datasets.

show abstract

Channel Redundancy and Overlap in Convolutional Neural Networks with Channel-Wise NNK Graphs

Cited by 7 publications

References 23 publications

The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning

The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning

Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters

Study of Manifold Geometry Using Multiscale Non-Negative Kernel Graphs

Contact Info

Product

Resources

About