Meta-learning via Language Model In-context Tuning

Chen, Yanda; Zhong, Ruiqi; Zha, Sheng; Karypis, George; He, He

doi:10.18653/v1/2022.acl-long.53

Cited by 16 publications

(10 citation statements)

References 40 publications

(46 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, using humans to create open-domain instruction datasets like OpenAI did will encounter the following challenges. The whole annotating process is extremely expensive and time-consuming [18][19][20][21]. On the other hand, the difficulty level distribution of human-created instructions is skewed towards being easy or moderate, with fewer difficult ones (according to the difficulty statistics of ShareGPT [22] from Figure 7a).…”

Section: Instruction Learningmentioning

confidence: 99%

Geography of Technology Transfer in China

Liu

2023

East China Normal University Scientific Reports

View full text Add to dashboard Cite

Graphs can model complex relationships between objects, enabling a myriad of Web applications such as online page/article classification and social recommendation. While graph neural networks (GNNs) have emerged as a powerful tool for graph representation learning, in an end-to-end supervised setting, their performance heavily relies on a large amount of task-specific supervision. To reduce labeling requirement, the "pre-train, fine-tune" and "pre-train, prompt" paradigms have become increasingly common. In particular, prompting is a popular alternative to fine-tuning in natural language processing, which is designed to narrow the gap between pre-training and downstream objectives in a task-specific manner. However, existing study of prompting on graphs is still limited, lacking a universal treatment to appeal to different downstream tasks. In this paper, we propose GraphPrompt, a novel pre-training and prompting framework on graphs. GraphPrompt not only unifies pre-training and downstream tasks into a common task template, but also employs a learnable prompt to assist a downstream task in locating the most relevant knowledge from the pre-trained model in a task-specific manner. Finally, we conduct extensive experiments on five public datasets to evaluate and analyze GraphPrompt. CCS CONCEPTS• Computing methodologies → Learning latent representations; • Information systems → Data mining.

show abstract

Section: Instruction Learningmentioning

confidence: 99%

Geography of Technology Transfer in China

Liu

2023

East China Normal University Scientific Reports

View full text Add to dashboard Cite

show abstract

“…ProtEx is also related to in-context tuning methods for few-shot tasks [43, 13], where pretrained language models are meta-trained to make predictions given an input and task-relevant exemplars. These works show strong performance on unseen tasks, enabled by the LM’s ability to make predictions from an input and a few in-context exemplars.…”

Section: Background and Related Workmentioning

confidence: 99%

ProtEx: A Retrieval-Augmented Approach for Protein Function Prediction

Shaw,

Gurram,

Belanger

et al. 2024

Preprint

View full text Add to dashboard Cite

Mapping a protein sequence to its underlying biological function is a critical problem of increasing importance in biology. In this work, we propose ProtEx, a retrieval-augmented approach for protein function prediction that leverages exem-plars from a database to improve accuracy and robustness and enable generalization to unseen classes. Our approach relies on a novel multi-sequence pretraining task, and a fine-tuning strategy that effectively conditions predictions on retrieved ex-emplars. Our method achieves state-of-the-art results across multiple datasets and settings for predicting Enzyme Commission (EC) numbers, Gene Ontology (GO) terms, and Pfam families. Our ablations and analysis highlight the impact of conditioning predictions on exemplar sequences, especially for classes and sequences less well represented in the training data.

show abstract

“…Additionally, researchers should evaluate the integrity of the labeling process (i.e., the process that determines which items belong to which labels (e.g., Mirończuk & Protasiewicz, 2018). If mislabeled, scale items are likely to hurt model performance (e.g., Chen et al, 2022; Phang et al, 2019; Saarikoski et al, 2015; Schick & Schütze, 2021). In the same vein, researchers may be motivated to include items that are indirectly related to the dimension labels of interest to obtain a larger number of items for training (e.g., collecting popular scales used in clinical psychology and labeling them as “neuroticism” items or collecting “extraversion” items from leadership scales).…”

Section: Demonstration: Training Transformers To Classify Personality...mentioning

confidence: 99%

“Transforming” Personality Scale Development: Illustrating the Potential of State-of-the-Art Natural Language Processing

Jia

Lee

Kaplan

2023

Organizational Research Methods

View full text Add to dashboard Cite

Natural language processing (NLP) techniques are becoming increasingly popular in industrial and organizational psychology. One promising area for NLP-based applications is scale development; yet, while many possibilities exist, so far these applications have been restricted—mainly focusing on automated item generation. The current research expands this potential by illustrating an NLP-based approach to content analysis, which manually categorizes scale items by their measured constructs. In NLP, content analysis is performed as a text classification task whereby a model is trained to automatically assign scale items to the construct that they measure. Here, we present an approach to text classification—using state-of-the-art transformer models—that builds upon past approaches. We begin by introducing transformer models and their advantages over alternative methods. Next, we illustrate how to train a transformer to content analyze Big Five personality items. Then, we compare the models trained to human raters, finding that transformer models outperform human raters and several alternative models. Finally, we present practical considerations, limitations, and future research directions.

show abstract

Meta-learning via Language Model In-context Tuning

Cited by 16 publications

References 40 publications

Geography of Technology Transfer in China

Geography of Technology Transfer in China

ProtEx: A Retrieval-Augmented Approach for Protein Function Prediction

“Transforming” Personality Scale Development: Illustrating the Potential of State-of-the-Art Natural Language Processing

Contact Info

Product

Resources

About