SSN: Learning Sparse Switchable Normalization via SparsestMax

Shao, Wei; Meng, Tianjian; Li, Jingyu; Zhang, Ruimao; Li, Yudian; Wang, Xiaogang; Luo, Ping

doi:10.48550/arxiv.1903.03793

Cited by 6 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Normalization techniques in deep neural networks are originally designed for regularizing trained models and improving their generalization performance. Various normalization techniques [1,3,7,13,15,18,20,21,23,25] have been studied actively in recent years. The most popular technique is batch normalization (BN) [7], which normalizes activations over individual channels using data in a mini-batch while instance normalization (IN) [23] performs the same operation per instance instead of mini-batch.…”

Section: Normalization In Neural Networkmentioning

confidence: 99%

See 1 more Smart Citation

Learning to Optimize Domain Specific Normalization for Domain Generalization

Seo¹,

Suh²,

Kim³

et al. 2019

Preprint

View full text Add to dashboard Cite

We propose a simple but effective multi-source domain generalization technique based on deep neural networks by incorporating optimized normalization layers specific to individual domains. Our approach employs multiple normalization methods while learning a separate affine parameter per domain. For each domain, the activations are normalized by a weighted average of multiple normalization statistics. The normalization statistics are kept track of separately for each normalization type if necessary. Specifically, we employ batch and instance normalizations in our implementation and attempt to identify the best combination of two normalization methods in each domain and normalization layer. In addition, we augment new domains through the combinations of multiple existing domains to increase the diversity of source domains available during training. The optimized normalization layers and the domain augmentation are effective to enhance the generalizability of the learned model. We demonstrate the state-ofthe-art accuracy of our algorithm in the standard benchmark datasets.

show abstract

Section: Normalization In Neural Networkmentioning

confidence: 99%

“…Recently, batch-instance normalization (BIN) [18], switchable normalization (SN) [13], and sparse switchable normalization (SSN) [21] employ the combinations of multiple normalization types to maximize the benefit. Note that BIN considers batch and instance normalizations while SN uses LN additionally.…”

Section: Normalization In Neural Networkmentioning

confidence: 99%

Learning to Optimize Domain Specific Normalization for Domain Generalization

Seo¹,

Suh²,

Kim³

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…To combine the advantages of multiple normalization techniques, Switchable Normalization (SN) (Luo et al, 2018) and Sparse Switchable Normalization (SSN) (Shao et al, 2019) make use of differentiable learning to switch among different normalization methods.…”

Section: Related Workmentioning

confidence: 99%

Cross-Iteration Batch Normalization

Yao

Cao

Zheng

et al. 2021

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

View full text Add to dashboard Cite

A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique. Code is available at https://github.com/Howal/ Cross-iterationBatchNorm.

show abstract

“…For example, SN [25] computes BN, IN, and LN at the same time and uses Au-toML [23] to determine how to combine them. SSN [39] uses SparsestMax to get sparse SN. DN [27] proposes a more flexible form to represent normalizations and finds better normalizations.…”

Section: Related Workmentioning

confidence: 99%

Rethinking Normalization and Elimination Singularity in Neural Networks

Qiao,

Wang,

Liu

et al. 2019

Preprint

View full text Add to dashboard Cite

In this paper, we study normalization methods for neural networks from the perspective of elimination singularity. Elimination singularities correspond to the points on the training trajectory where neurons become consistently deactivated. They cause degenerate manifolds in the loss landscape which will slow down training and harm model performances. We show that channel-based normalizations (e.g. Layer Normalization and Group Normalization) are unable to guarantee a far distance from elimination singularities, in contrast with Batch Normalization which by design avoids models from getting too close to them. To address this issue, we propose Batch-Channel Normalization (BCN), which uses batch knowledge to avoid the elimination singularities in the training of channel-normalized models. Unlike Batch Normalization, BCN is able to run in both large-batch and microbatch training settings. The effectiveness of BCN is verified on many tasks, including image classification, object detection, instance segmentation, and semantic segmentation. The code is here: https://github.com/ joe-siyuan-qiao/Batch-Channel-Normalization.

show abstract

SSN: Learning Sparse Switchable Normalization via SparsestMax

Cited by 6 publications

References 14 publications

Learning to Optimize Domain Specific Normalization for Domain Generalization

Learning to Optimize Domain Specific Normalization for Domain Generalization

Cross-Iteration Batch Normalization

Rethinking Normalization and Elimination Singularity in Neural Networks

Contact Info

Product

Resources

About