Scientific Keyphrase Identification and Classification by Pre-Trained Language Models Intermediate Task Transfer Learning

Abstract

Scientific keyphrase identification and classification is the task of detecting and classifying keyphrases from scholarly text with their types from a set of predefined classes. This task has a wide range of benefits, but it is still challenging in performance due to the lack of large amounts of labeled data required for training deep neural models. In order to overcome this challenge, we explore pre-trained language models BERT and SciBERT with intermediate task transfer learning, using 42 data-rich related intermediate-target task combinations. We reveal that intermediate task transfer learning on SciBERT induces a better starting point for target task fine-tuning compared with BERT and achieves competitive performance in scientific keyphrase identification and classification compared to both previous works and strong baselines. Interestingly, we observe that BERT with intermediate task transfer learning fails to improve the performance of scientific keyphrase identification and classification potentially due to significant catastrophic forgetting. This result highlights that scientific knowledge achieved during the pre-training of language models on large scientific collections plays an important role in the target tasks. We also observe that sequence tagging related intermediate tasks, especially syntactic structure learning tasks such as POS Tagging, tend to work best for scientific keyphrase identification and classification.

Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training

“…Ye et al (2021b) propose to dynamically align target phrases to eliminate the influence of order, as highlighted by Meng et al (2021). Mu et al (2020); Liu et al (2020a); Park and Caragea (2020) use pre-trained language models for better representations of documents.…”

Pre-training Methods in Information Retrieval

“…It can not only provide users a quick view of result documents (similar to docu-ment summarization) but may also benefit downstream tasks such as document indexing, document recommendation, and query suggestion. Most of them formulated keyphrase extraction as a sequential labeling task (Lim et al, 2020;Wu et al, 2021;Park and Caragea, 2020;Sahrawat et al, 2020;Liu et al, 2020). For example, some work (Sahrawat et al, 2020;Park and Caragea, 2020) adopted contextualized embeddings generated by BERT or SciBERT (Beltagy et al, 2019) as the input of their BiLSTM-CRF architecture for scientific keyphrase extraction.…”

“…Most of them formulated keyphrase extraction as a sequential labeling task (Lim et al, 2020;Wu et al, 2021;Park and Caragea, 2020;Sahrawat et al, 2020;Liu et al, 2020). For example, some work (Sahrawat et al, 2020;Park and Caragea, 2020) adopted contextualized embeddings generated by BERT or SciBERT (Beltagy et al, 2019) as the input of their BiLSTM-CRF architecture for scientific keyphrase extraction. Tang et al (2019) used BERT with attention layer to automatically extract keywords from the clinical notes.…”

Studying the Evolution of Scientific Topics and their Relationships

“…(topic t , type c ), typec ∈ {task, algorithm, data}, topict ∈ T, with T representing the list of topics generated by the Latent Dirichlet Allocation model (LDA) (Blei et al, 2003). These topic categories can be useful beyond our field and application, for example in question answering systems or paper recommendation systems (Augenstein et al, 2017;Park and Caragea, 2020;Luan et al, 2018;QasemiZadeh and Schumann, 2016). In order to identify the topics occurring in our corpus of scientific texts, we first train an LDA model on the full texts extracted from computational linguistics articles, and use it to extract a set of 100 topics which we will use to analyze the evolution of the field in the next stages of our study.…”