When less is more: Simplifying inputs aids neural network understanding

Schirrmeister, Robin Tibor; Liu, Rosanne; Hooker, Sara; Ball, Tonio

doi:10.48550/arxiv.2201.05610

Cited by 3 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Devising strategies for data pruning and constructing optimal subsets is a recent topic of interest in the area of optimization and active learning (Dong et al, 2019;Kaushal et al, 2019;Saadatfar et al, 2020;Durga et al, 2021;Kothawade et al, 2021;Killamsetty et al, 2021;Paul et al, 2021;Kothyari et al, 2021;Ahia et al, 2021). A few studies have examined the training landscape for drawing clues about the optimal subset creation (Toneva et al, 2018;Agarwal et al, 2020;Baldock et al, 2021;Paul et al, 2021;Schirrmeister et al, 2022). The work on coresets (Tolochinsky & Feldman, 2018;Huang et al, 2021;Jiang et al, 2021;Jubran et al, 2021;Mirzasoleiman et al, 2020) is also being actively researched in the regime of optimization and active learning.…”

Section: Related Workmentioning

confidence: 99%

Towards Representative Subset Selection for Self-Supervised Speech Recognition

Hameed¹,

Qazi²,

Raza³

2022

Preprint

View full text Add to dashboard Cite

Self-supervised speech recognition models require considerable labeled training data for learning high-fidelity representations for Automatic Speech Recognition (ASR), which hinders their application to low-resource languages. We consider the task of identifying an optimal subset of training data to fine-tune self-supervised speech models for ASR. We make a surprising observation that active learning strategies for sampling harder-to-learn examples do not perform better than random subset selection for fine-tuning selfsupervised ASR. We then present the COWER-AGE algorithm for better subset selection in selfsupervised ASR which is based on our finding that ensuring the coverage of examples based on training WER in the early training epochs leads to better generalization performance. Extensive experiments on the wav2vec 2.0 model and TIMIT dataset show the effectiveness of COWERAGE, with up to 27% absolute WER improvement over active learning methods. We also report the connection between training WER and the phonemic cover and demonstrate that our algorithm ensures inclusion of phonemically diverse examples.

show abstract

Section: Related Workmentioning

confidence: 99%

Towards Representative Subset Selection for Self-Supervised Speech Recognition

Hameed¹,

Qazi²,

Raza³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…(Vicol et al, 2022) studies the implicit bias between warm start vs. cold start of the bilevel optimization under the meta-model matching dataset distillation approaches. (Schirrmeister et al, 2022) shows that dataset distillation methods can be regularized towards a simpler dataset using a pre-trained generative model. (Maalouf et al, 2023) provides theoretical support for the existence of small distilled dataset in the context of kernel ridge regression models.…”

Section: Introductionmentioning

confidence: 99%

Towards fairer datasets

Yang

Qinami

et al. 2020

Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency

189

View full text Add to dashboard Cite

Dataset distillation has emerged as a strategy to overcome the hurdles associated with large datasets by learning a compact set of synthetic data that retains essential information from the original dataset. While distilled data can be used to train high performing models, little is understood about how the information is stored. In this study, we posit and answer three questions about the behavior, representativeness, and point-wise information content of distilled data. We reveal distilled data cannot serve as a substitute for real data during training outside the standard evaluation setting for dataset distillation. Additionally, the distillation process retains high task performance by compressing information related to the early training dynamics of real models. Finally, we provide an framework for interpreting distilled data and reveal that individual distilled data points contain meaningful semantic information. This investigation sheds light on the intricate nature of distilled data, providing a better understanding on how they can be effectively utilized.

show abstract

Neural network structure simplification by assessing evolution in node weight magnitude

Riedel,

Segev

2023

Mach Learn

View full text Add to dashboard Cite

When less is more: Simplifying inputs aids neural network understanding

Cited by 3 publications

References 0 publications

Towards Representative Subset Selection for Self-Supervised Speech Recognition

Towards Representative Subset Selection for Self-Supervised Speech Recognition

Towards fairer datasets

Neural network structure simplification by assessing evolution in node weight magnitude

Contact Info

Product

Resources

About