Intracranial Error Detection via Deep Learning

Volker, Martin A.; Hammer, Jiří; Schirrmeister, Robin Tibor; Behncke, Joos; Fiederer, Lukas Dominique Josef; Schulze‐Bonhage, Andreas; Marusič, Petr; Burgard, Wolfram; Ball, Tonio

doi:10.48550/arxiv.1805.01667

Cited by 1 publication

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, used AdamW to train a novel architecture for face detection on the standard WIDER FACE dataset (Yang et al, 2016), obtaining almost 10x faster predictions than the previous state of the art algorithms while achieving comparable performance. Völker et al (2018) employed AdamW with cosine annealing to train convolutional neural networks to classify and characterize error-related brain signals measured from intracranial electroencephalography (EEG) recordings.…”

Section: Use Of Adamw On Other Datasets and Architecturesmentioning

confidence: 99%

Decoupled Weight Decay Regularization

Loshchilov,

Hutter

2017

Preprint

2,194

1,926

View full text Add to dashboard Cite

L 2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L 2 regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is

show abstract

Section: Use Of Adamw On Other Datasets and Architecturesmentioning

confidence: 99%