Automatic Node Selection for Deep Neural Networks using Group Lasso Regularization

Ochiai, Tsubasa; Matsuda, Shigeki; Watanabe, H.; Katagiri, Shigeru

doi:10.48550/arxiv.1611.05527

Search citation statements

Order By: Relevance

Paper Sections

Select...

Related Work1

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2017

Publication Types

Select...

Other1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

(1 citation statement)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, [33] describe applications of structured sparsity not only for optimizing filters and layers but also computation time. Closest to our work in spirit, [24], [28] and [22] who use group sparsity to prune groups of weights-e.g. weights incident to a node.…”

Section: Related Workmentioning

confidence: 99%

Model Selection in Bayesian Neural Networks via Horseshoe Priors

Ghosh,

Doshi-Velez

2017

Preprint

View full text Add to dashboard Cite

Bayesian Neural Networks (BNNs) have recently received increasing attention for their ability to provide well-calibrated posterior uncertainties. However, model selection-even choosing the number of nodes-remains an open question. In this work, we apply a horseshoe prior over node preactivations of a Bayesian neural network, which effectively turns off nodes that do not help explain the data. We demonstrate that our prior prevents the BNN from under-fitting even when the number of nodes required is grossly over-estimated. Moreover, this model selection over the number of nodes doesn't come at the expense of predictive or computational performance; in fact, we learn smaller networks with comparable predictive performance to current approaches.

show abstract