Prototypical Calibration for Few-shot Learning of Language Models

Abstract

In-context learning of GPT-like models has been recognized as fragile across different hand-crafted templates, and demonstration permutations. In this work, we propose prototypical calibration to adaptively learn a more robust decision boundary for zero-and few-shot classification, instead of greedy decoding. Concretely, our method first adopts Gaussian mixture distribution to estimate the prototypical clusters for all categories. Then we assign each cluster to the corresponding label by solving a weighted bipartite matching problem. Given an example, its prediction is calibrated by the likelihood of prototypical clusters. Experimental results show that prototypical calibration yields a 15% absolute improvement on a diverse set of tasks. Extensive analysis across different scales also indicates that our method calibrates the decision boundary as expected, greatly improving the robustness of GPT to templates, permutations, and class imbalance.

Generative Calibration for In-context Learning

“…In short summary, in-context models are mainly biased by shifting the label marginal, which is generally implicitly assumed in previous works (Zhao et al, 2021;Han et al, 2022;Fei et al, 2023). However, to the best of our knowledge, we are the first to verify this systematically.…”

“…The data label marginal q(y) is hard to estimate since we only have a few training examples in our setting. Previous works (Zhao et al, 2021;Min et al, 2021;Han et al, 2022;Fei et al, 2023) typically assume q(y) to be uniform, yielding the following classifier:…”

“…This instability motivates lots of efforts on searching for a better prompt in terms of better few-shot training examples (Rubin et al, 2021;Liu et al, 2021;Su et al, 2022;, better order permutation of those examples (Lu et al, 2021;, and slot position in the template (Holtzman et al, 2021;Min et al, 2021). Another promising direction is calibration (Zhao et al, 2021;Han et al, 2022;Fei et al, 2023) that adjusts the ICL predictive distribution via an estimated bias. Compared with prompt optimization methods, such a stream is usually much more lightweight and gains substantial and consistency improvement, while also reducing the instability.…”

“…Actually, most previous state-of-the-art calibration approaches (Zhao et al, 2021;Han et al, 2022;Fei et al, 2023) implicitly assume that the ICL predictive distribution has a shift on label marginal. However, none of them validate the soundness of that assumption, while we for the first time verify this systematically.…”

A Foreground Prototype-Based One-Shot Segmentation of Brain Tumors

“…The use of autoencoders in NVDLMED [ 8 ] for segmentation makes it more computationally expensive. In comparison to these models [ 41 ], our proposed approach contains fewer parameters and is not much computationally expensive. Furthermore, our one-shot segmentation model took a very low inference time for each image, which makes it very suitable to test unseen images in clinical settings.…”