Anchor Loss: Modulating Loss Scale Based on Prediction Difficulty

Ryou, Serim; Jeong, Shinkyu; Perona, Pietro

doi:10.1109/iccv.2019.00609

Cited by 49 publications

(22 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, [18] scaled the loss by inverse class frequency. An alternative strategy down-weighs the loss of well-classified examples, preventing easy negatives from dominating the loss [27] or dynamically rescale the cross-entropy loss based on the difficulty to classify a sample [34]. [6] proposed to encourage larger margins for rare classes.…”

Section: Long-tail Recognitionmentioning

confidence: 99%

Distributional Robustness Loss for Long-tail Learning

Samuel

Chechik

2021

Preprint

View full text Add to dashboard Cite

Real-world data is often unbalanced and long-tailed, but deep models struggle to recognize rare classes in the presence of frequent classes. To address unbalanced data, most studies try balancing the data, the loss, or the classifier to reduce classification bias towards head classes. Far less attention has been given to the latent representations learned with unbalanced data. We show that the feature extractor part of deep networks suffers greatly from this bias. We propose a new loss based on robustness theory, which encourages the model to learn high-quality representations for both head and tail classes. While the general form of the robustness loss may be hard to compute, we further derive an easy-to-compute upper bound that can be minimized efficiently. This procedure reduces representation bias towards head classes in the feature space and achieves new SOTA results on CIFAR100-LT, ImageNet-LT, and iNaturalist long-tail benchmarks. We find that training with robustness increases recognition accuracy of tail classes while largely maintaining the accuracy of head classes. The new robustness loss can be combined with various classifier balancing techniques and can be applied to representations at several layers of the deep model.

show abstract

Section: Long-tail Recognitionmentioning

confidence: 99%

Distributional Robustness Loss for Long-tail Learning

Samuel

Chechik

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The similar problem arises in the field of object detection where the model should detect scarce objects from countless of easily classified background examples. This problem is typically addressed via adjusting the loss scales based on example difficulty and avoiding major gradient updates on trivial predictions [14,15].…”

Section: Overlap Ratio Modulationmentioning

confidence: 99%

Time-Domain Loss Modulation Based on Overlap Ratio for Monaural Conversational Speaker Separation

Taherian

Wang

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Existing speaker separation methods deliver excellent performance on fully overlapped signal mixtures. To apply these methods in daily conversations that include occasional concurrent speakers, recent studies incorporate both overlapped and non-overlapped segments in the training data. However, such training data can degrade the separation performance due to triviality of non-overlapped segments where the model reflects the input to the output. We propose a new loss function for speaker separation based on permutation invariant training that dynamically reweighs losses using the segment overlap ratio. The new loss function emphasizes overlapped regions while deemphasizing the segments with single speakers. We demonstrate the effectiveness of the proposed loss function on an automatic speech recognition (ASR) task. Experiments on the recently introduced LibriCSS corpus show that our proposed single-channel method produces consistent improvements compared to baseline methods.

show abstract

“…This is because the background occupies most of the image in an objectdetection problem, and it is necessary to eliminate the imbalance of the label to identify a specific object in the minority class. Thus far, studies segmented medical images using a loss function based on the Dice coefficient [43,44], and robust losses for imbalanced data, such as focal loss [18], dice loss [19], and anchor loss [20] were proposed. We applied BERT, adopting focal loss and dice loss as the loss functions, to the text segmentation of novels into paragraphs, and demonstrated the effectiveness of the approach [45][46][47].…”

Section: Imbalanced Classificationmentioning

confidence: 99%

“…Motivated by focal loss, anchor loss (AL) [20] is a loss function that dynamically scales CE loss on the basis of the difficulty of predicting the sample. Similar to focal loss, AL was proposed for use in an object-detection task where the imbalance between the number of pixels of the background and the target object is a bottleneck.…”

Section: Anchor Lossmentioning

confidence: 99%

“…However, because the number of paragraphs is small relative to the number of sentences, it is necessary to consider the imbalance in the number of the data. Therefore, we used focal loss [18], dice loss [19], and anchor loss [20], which were confirmed to be robust to imbalanced classification, as a loss function of BERT, which is highly accurate in various naturallanguage processing (NLP) tasks. Through experiments, we confirmed that the novel could be divided into appropriate paragraphs by using our method.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Paragraph Boundary Recognition in Novels for Story Understanding

Iikura¹,

Okada²,

Mori³

2021

Applied Sciences

View full text Add to dashboard Cite

The understanding of narrative stories by computer is an important task for their automatic generation. To date, high-performance neural-network technologies such as BERT have been applied to tasks such as the Story Cloze Test and Story Completion. In this study, we focus on the text segmentation of novels into paragraphs, which is an important writing technique for readers to deepen their understanding of the texts. This type of segmentation, which we call “paragraph boundary recognition”, can be considered to be a binary classification problem in terms of the presence or absence of a boundary, such as a paragraph between target sentences. However, in this case, the data imbalance becomes a bottleneck because the number of paragraphs is generally smaller than the number of sentences. To deal with this problem, we introduced several cost-sensitive loss functions, namely. focal loss, dice loss, and anchor loss, which were robust for imbalanced classification in BERT. In addition, introducing the threshold-moving technique into the model was effective in estimating paragraph boundaries. As a result of the experiment on three newly created datasets, BERT with dice loss and threshold moving obtained a higher F1 than the original BERT had using cross-entropy loss as its loss function (76% to 80%, 50% to 54%, 59% to 63%).

show abstract

Anchor Loss: Modulating Loss Scale Based on Prediction Difficulty

Cited by 49 publications

References 36 publications

Distributional Robustness Loss for Long-tail Learning

Distributional Robustness Loss for Long-tail Learning

Time-Domain Loss Modulation Based on Overlap Ratio for Monaural Conversational Speaker Separation

Paragraph Boundary Recognition in Novels for Story Understanding

Contact Info

Product

Resources

About