The Devil is in the Margin: Margin-based Label Smoothing for Network Calibration

Calibrating segmentation networks with margin-based label smoothing

“…Note that the reasoning behind working directly on the logit space is two-fold. First, observations in [11] suggest that directly imposing the constraints on the logits results in better performance than in the softmax predictions. And second, by imposing a bounded constraint on the logits values 2 , their magnitudes are further decreased, which has a favorable effect on model calibration [17].…”

“…And second, by imposing a bounded constraint on the logits values 2 , their magnitudes are further decreased, which has a favorable effect on model calibration [17]. We stress that despite both [11] and our method enforce constraints on the predicted logits, [11] is fundamentally different. In particular, [11] imposes an inequality constraint on the logit distances so that it encourages uniform-alike distributions up to a given margin, disregarding the importance of each class in a given patch.…”

“…Furthermore, findings from recent works on calibration [16,17] have demonstrated that popular classification losses, such as Label Smoothing (LS) [22] and Focal Loss (FL) [10], have a favorable effect on model calibration, as they implicitly integrate an entropy maximization objective. Following these works, [11,18] presented a unified view of state-of-the-art calibration approaches [21,22,10] showing that these strategies can be viewed as approximations of a linear penalty imposing equality constraints on logit distances. The associated equality constraint results in gradients that continually push towards a non-informative solution, potentially hindering the ability to achieve the optimal balance between discriminative performance and model calibration.…”

“…The associated equality constraint results in gradients that continually push towards a non-informative solution, potentially hindering the ability to achieve the optimal balance between discriminative performance and model calibration. To alleviate this limitation, [11,18] proposed a simple and flexible alternative based on inequality constraints, which imposes a controllable margin on logit distances. Despite the progress brought by these methods, none of them explicitly considers pixel relationships, which is fundamental in the context of image segmentation.…”

Rethinking Confidence Calibration for Failure Prediction

“…Recently, many approaches have been developed to alleviate the overconfidence problem by calibrating the confidence, i.e., matching the accuracy and confidence scores to reflect the predictive uncertainty [46]. Specifically, one category of approaches [22,43,50,51,57,65,70,72,74,79] aim to learn well-calibrated models during training. For instance, mixup [65], label smoothing [51] and focal loss [50] have been demonstrated to be effective for confidence calibration.…”