Post-training discriminative pruning for RBMs

Sánchez-Gutiérrez, Máximo Eduardo; Albornoz, Enrique Marcelo; Rufiner, Hugo Leonardo; Close, John Goddard

doi:10.1007/s00500-017-2784-3

Cited by 6 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Pruning studies in artificial neural networks often estimate parameter importance with respect to the curvature of a cost function on network performance or discriminative power in supervised learning problems [23,25,50]. This requires access to a global state, and is not applicable to biological networks.…”

Section: Biological Plausibility Of Different Pruning Criteriamentioning

confidence: 99%

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

2021

View full text Add to dashboard Cite

During development, biological neural networks produce more synapses and neurons than needed. Many of these synapses and neurons are later removed in a process known as neural pruning. Why networks should initially be over-populated, and the processes that determine which synapses and neurons are ultimately pruned, remains unclear. We study the mechanisms and significance of neural pruning in model neural networks. In a deep Boltzmann machine model of sensory encoding, we find that (1) synaptic pruning is necessary to learn efficient network architectures that retain computationally-relevant connections, (2) pruning by synaptic weight alone does not optimize network size and (3) pruning based on a locally-available measure of importance based on Fisher information allows the network to identify structurally important vs. unimportant connections and neurons. This locally-available measure of importance has a biological interpretation in terms of the correlations between presynaptic and postsynaptic neurons, and implies an efficient activity-driven pruning rule. Overall, we show how local activity-dependent synaptic pruning can solve the global problem of optimizing a network architecture. We relate these findings to biology as follows: (I) Synaptic over-production is necessary for activity-dependent connectivity optimization. (II) In networks that have more neurons than needed, cells compete for activity, and only the most important and selective neurons are retained. (III) Cells may also be pruned due to a loss of synapses on their axons. This occurs when the information they convey is not relevant to the target population.

show abstract

Section: Biological Plausibility Of Different Pruning Criteriamentioning

confidence: 99%

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

2021

View full text Add to dashboard Cite

show abstract

“…Pruning studies in arti cial neural networks often estimate parameter importance with respect to the curvature of a cost function on network performance or discriminative power in supervised learning problems (e.g. LeCun et al, 1990;Sánchez-Gutiérrez et al, 2019). This requires access to global state, and is not applicable to biological neural networks.…”

Section: Biological Plausibility Of DI Erent Pruning Criteriamentioning

confidence: 99%

The Information Theory of Developmental Pruning: Optimizing Global Network Architecture Using Local Synaptic Rules

Scholl¹,

Rule

Hennig

2020

Preprint

View full text Add to dashboard Cite

During development, biological neural networks produce more synapses and neurons than needed. Many of these synapses and neurons are later removed in a process known as neural pruning. Why networks should initially be over-populated, and processes that determine which synapses and neurons are ultimately pruned, remains unclear. We study the mechanisms and significance of neural pruning in model neural network. In a deep Boltzmann machine model of sensory encoding, we find that (1) synaptic pruning is necessary to learn efficient network architectures that retain computationally-relevant connections, (2) pruning by synaptic weight alone does not optimize network size and (3) pruning based on a locally-available proxy for "sloppiness" based on Fisher Information allows the network to identify structurally important vs. unimportant connections and neurons. This locally-available measure of importance has a biological interpretation in terms of the correlations between presynaptic and postsynaptic neurons, and implies an efficient activity-driven pruning rule. Overall, we show how local activity-dependent synaptic pruning can solve the global problem of optimizing a network architecture. We relate these findings to biology as follows: (I) Synaptic over-production is necessary for activity-dependent connectivity optimization. (II) In networks that have more neurons than needed, cells compete for activity, and only the most important and selective neurons are retained. (III) Cells may also be pruned due to a loss of synapses on their axons. This occurs when the information they convey is not relevant to the target population.

show abstract

“…Pattern recognition is a discipline which is mainly oriented to the generation of algorithms or methods that can decide an action based upon certain recognized similarities (patterns) in the input data. Within signal classification, which is perhaps one of the most important subfields of pattern recognition, several discrepancy measures have been used in problems coming from a wide variety of areas such as machine learning [1], image and speech processing [2], neural networks [3], and biomedical signal processing [4,5]. Among them, the most commonly used is probably the Kullback-Leibler (KL) divergence [6,7].…”

Section: Introductionmentioning

confidence: 99%

Complexity‐Based Discrepancy Measures Applied to Detection of Apnea‐Hypopnea Events

et al. 2018

View full text Add to dashboard Cite

In recent years, an increasing interest in the development of discriminative methods based on sparse representations with discrete dictionaries for signal classification has been observed. It is still unclear, however, what is the most appropriate way for introducing discriminative information into the sparse representation problem. It is also unknown which is the best discrepancy measure for classification purposes. In the context of feature selection problems, several complexity-based measures have been proposed. The main objective of this work is to explore a method that uses such measures for constructing discriminative subdictionaries for detecting apnea-hypopnea events using pulse oximetry signals. Besides traditional discrepancy measures, we study a simple one called Difference of Conditional Activation Frequency (DCAF). We additionally explore the combined effect of overcompleteness and redundancy of the dictionary as well as the sparsity level of the representation. Results show that complexity-based measures are capable of adequately pointing out discriminative atoms. Particularly, DCAF yields competitive averaged detection accuracy rates of 72.57% at low computational cost. Additionally, ROC curve analyses show averaged diagnostic sensitivity and specificity of 81.88% and 87.32%, respectively. This shows that discriminative subdictionary construction methods for sparse representations of pulse oximetry signals constitute a valuable tool for apnea-hypopnea screening.

show abstract

Post-training discriminative pruning for RBMs

Cited by 6 publications

References 39 publications

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

The information theory of developmental pruning: Optimizing global network architectures using local synaptic rules

The Information Theory of Developmental Pruning: Optimizing Global Network Architecture Using Local Synaptic Rules

Complexity‐Based Discrepancy Measures Applied to Detection of Apnea‐Hypopnea Events

Contact Info

Product

Resources

About