What’s Hidden in a Randomly Weighted Neural Network?

Ramanujan, Vivek; Wortsman, Mitchell; Kembhavi, Aniruddha; Farhadi, Ali; Rastegari, Mohammad

doi:10.1109/cvpr42600.2020.01191

Cited by 143 publications

(271 citation statements)

References 8 publications

Supporting

Mentioning

237

Contrasting

Order By: Relevance

“…It was reported that selective tunings, such as number-selective responses, can emerge from the multiplication of random matrices ( 23 ) and that the structure of a randomly initialized convolutional neural network can provide a priori information about the low-level statistics in natural images, enabling the reconstruction of the corrupted images without any training for feature extraction ( 24 ). Furthermore, a recent study showed that subnetworks from randomly initialized neural networks can perform image classification ( 25 ), implying the ability of a randomly initialized network to engage in visual feature extraction.…”

Section: Introductionmentioning

confidence: 99%

Visual number sense in untrained deep neural networks

et al. 2021

View full text Add to dashboard Cite

Number sense, the ability to estimate numerosity, is observed in naïve animals, but how this cognitive function emerges in the brain remains unclear. Here, using an artificial deep neural network that models the ventral visual stream of the brain, we show that number-selective neurons can arise spontaneously, even in the complete absence of learning. We also show that the responses of these neurons can induce the abstract number sense, the ability to discriminate numerosity independent of low-level visual cues. We found number tuning in a randomly initialized network originating from a combination of monotonically decreasing and increasing neuronal activities, which emerges spontaneously from the statistical properties of bottom-up projections. We confirmed that the responses of these number-selective neurons show the single- and multineuron characteristics observed in the brain and enable the network to perform number comparison tasks. These findings provide insight into the origin of innate cognitive functions.

show abstract

Section: Introductionmentioning

confidence: 99%

Visual number sense in untrained deep neural networks

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Empirical evidence suggests the likely existence ofN . [5] and [6] showed that an untrained random model simultaneously contains sub-networks that perform well. At specific sparsity levels, these models perform as well as individually trained dense models.…”

Section: Model Formulationmentioning

confidence: 99%

“…To deal with this issue, model pruning methods have been developed to remove unimportant connections in weight matrices of neural network models [29,30,1]. The resulting pruned models contain only sparse structures, allowing them to run efficiently at [31,32,5,6].…”

Section: Related Workmentioning

confidence: 99%

“…Recently, [5] and [6] showed that untrained random networks contain sub-networks at arbitrary sparsity levels that perform well without training. The best of these sub-networks, usually at around 50% sparsity, can perform as well as the full (i.e.…”

Section: Related Workmentioning

confidence: 99%

“…This work was done while Zhaofeng Wu was an intern at Google. DSNN was inspired by recent work [5,6] which showed that even for untrained random networks, there exist arbitrarily sparse sub-networks that achieve very high quality. Therefore, it is likely that trained networks also simultaneously contain powerful subnetworks at multiple sparsity levels.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dynamic Sparsity Neural Networks for Automatic Speech Recognition

Zhao

Liang

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In automatic speech recognition (ASR), model pruning is a widely adopted technique that reduces model size and latency to deploy neural network models on edge devices with resource constraints. However, multiple models with different sparsity levels usually need to be separately trained and deployed to heterogeneous target hardware with different resource specifications and for applications that have various latency requirements. In this paper, we present Dynamic Sparsity Neural Networks (DSNN) that, once trained, can instantly switch to any predefined sparsity configuration at run-time. We demonstrate the effectiveness and flexibility of DSNN using experiments on internal production datasets with Google Voice Search data, and show that the performance of a DSNN model is on par with that of individually trained single sparsity networks. Our trained DSNN model, therefore, can greatly ease the training process and simplify deployment in diverse scenarios with resource constraints.

show abstract

Input Representation

2022

An Introduction to Audio Content Analysis

View full text Add to dashboard Cite

What’s Hidden in a Randomly Weighted Neural Network?

Cited by 143 publications

References 8 publications

Visual number sense in untrained deep neural networks

Visual number sense in untrained deep neural networks

Dynamic Sparsity Neural Networks for Automatic Speech Recognition

Input Representation

Contact Info

Product

Resources

About