Data Curation Alone Can Stabilize In-context Learning

Chang, Ting-Yun; Jia, Robin

doi:10.18653/v1/2023.acl-long.452

Cited by 2 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another set of approaches leverages reinforcement learning to select good samples Shum et al, 2023;Scarlatos & Lan, 2023). Finally, the Datamodels approach trains a linear model to predict the performance gain of a set of samples to select the subset that would lead to the highest possible performance increase (Ilyas et al, 2022;Chang & Jia, 2023;Jundi & Lapesa, 2022;Vilar et al, 2023).…”

Section: Related Workmentioning

confidence: 99%

“…Datamodels Selection draws inspiration from the Datamodels (Ilyas et al, 2022;Chang & Jia, 2023), but works on the level of the strategies instead of samples. First, a set of 150 random combinations of strategies is created and evaluated.…”

Section: Learnabilitymentioning

confidence: 99%

“…This also includes the use of larger validation or held-out set when selecting strategies. If we considered the true few-shot learning setting, only the active learning strategies (i.e., Entropy, Margin and Least Confidence) could be used, as they work with unlabelled data only, or a different approach for sample selection should be employed, such as using weak-labels to select the best samples, or assign every single class to all the labels and perform the sample selection that way to select both best performing samples as well as the class for them (as it is done in (Chang & Jia, 2023)).…”

Section: A Limitationsmentioning

confidence: 99%

“…Therefore, the model is limited by its max input length. At the same time, this approach was found to be sensitive to the selected samples in the support set and their order (Zhao et al, 2021;Pecher et al, 2023;Chang & Jia, 2023).…”

Section: B Few-shot Learning Problem Formulationmentioning

confidence: 99%

“…Even though a lot of research is dedicated to sample selection, such as active learning (Ren et al, 2021) or core-set selection (Guo et al, 2022), the most common approach in few-shot learning is a simple random selection. The exception is in-context learning, where a large number of sample selection strategies have been proposed recently, selecting samples according to their properties such as similarity, diversity, informativeness or quality Chang & Jia, 2023).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

FireAnt: Claim-Based Medical Misinformation Detection and Monitoring

Pecher

Srba

Móro

et al. 2021

Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track

View full text Add to dashboard Cite

In few-shot learning, such as meta-learning, fewshot fine-tuning or in-context learning, the limited number of samples used to train a model have a significant impact on the overall success. Although a large number of sample selection strategies exist, their impact on the performance of few-shot learning is not extensively known, as most of them have been so far evaluated in typical supervised settings only. In this paper, we thoroughly investigate the impact of 20 sample selection strategies on the performance of 5 fewshot learning approaches over 8 image and 6 text datasets. In addition, we propose a new method for automatic combination of sample selection strategies (ACSESS) that leverages the strengths and complementary information of the individual strategies. The experimental results show that our method consistently outperforms the individual selection strategies, as well as the recently proposed method for selecting support examples for in-context learning. We also show a strong modality, dataset and approach dependence for the majority of strategies as well as their dependence on the number of shots -demonstrating that the sample selection strategies play a significant role for lower number of shots, but regresses to random selection at higher number of shots.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Learnabilitymentioning

confidence: 99%

Section: A Limitationsmentioning

confidence: 99%

Section: B Few-shot Learning Problem Formulationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

FireAnt: Claim-Based Medical Misinformation Detection and Monitoring

Pecher

Srba

Móro

et al. 2021

Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track

View full text Add to dashboard Cite

show abstract

Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis

Yang,

Kim,

Jang

et al. 2024

Transactions of the Association for Computational Linguistics

View full text Add to dashboard Cite

Previous work in prompt engineering for large language models has introduced different gradient-free probability-based prompt selection methods that aim to choose the optimal prompt among the candidates for a given task but have failed to provide a comprehensive and fair comparison between each other. In this paper, we propose a unified framework to interpret and evaluate the existing probability-based prompt selection methods by performing extensive experiments on 13 common and diverse NLP tasks. We find that each of the existing methods can be interpreted as some variant of the method that maximizes mutual information between the input and the predicted output (MI). Utilizing this finding, we develop several other combinatorial variants of MI and increase the effectiveness of the oracle prompt selection method from 87.79% to 94.98%, measured as the ratio of the performance of the selected prompt to that of the optimal oracle prompt. Furthermore, considering that all the methods rely on the output probability distribution of the model that might be biased, we propose a novel calibration method called Calibration by Marginalization (CBM) that is orthogonal to the existing methods and helps increase the prompt selection effectiveness of the best method to 96.85%, achieving 99.44% of the oracle prompt F1 without calibration.1

show abstract

Data Curation Alone Can Stabilize In-context Learning

Cited by 2 publications

References 0 publications

FireAnt: Claim-Based Medical Misinformation Detection and Monitoring

FireAnt: Claim-Based Medical Misinformation Detection and Monitoring

Improving Probability-based Prompt Selection Through Unified Evaluation and Analysis

Contact Info

Product

Resources

About