NTNU-1$@$ScienceIE at SemEval-2017 Task 10: Identifying and
            Labelling Keyphrases with Conditional Random Fields

Marsi, Erwin; Sikdar, Utpal Kumar; Marco, Cristina; Barik, Biswanath; Sætre, Rune

doi:10.18653/v1/s17-2162

Cited by 2 publications

(1 citation statement)

References 4 publications

(6 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Most of the earlier approaches to SKIC rely on hand-crafted linguistic features. For example, Lee et al (2017) and Marsi et al (2017) employ syntactic and semantic features such as Part-of-Speech tags and word lemmas, as input to Conditional Random Fields (CRFs). Liu et al (2017) formulate SKIC as a supervised multi-class classification problem.…”

Section: Related Workmentioning

confidence: 99%

Scientific Keyphrase Identification and Classification by Pre-Trained Language Models Intermediate Task Transfer Learning

Park¹,

Caragea²

2020

Proceedings of the 28th International Conference on Computational Linguistics

View full text Add to dashboard Cite

Scientific keyphrase identification and classification is the task of detecting and classifying keyphrases from scholarly text with their types from a set of predefined classes. This task has a wide range of benefits, but it is still challenging in performance due to the lack of large amounts of labeled data required for training deep neural models. In order to overcome this challenge, we explore pre-trained language models BERT and SciBERT with intermediate task transfer learning, using 42 data-rich related intermediate-target task combinations. We reveal that intermediate task transfer learning on SciBERT induces a better starting point for target task fine-tuning compared with BERT and achieves competitive performance in scientific keyphrase identification and classification compared to both previous works and strong baselines. Interestingly, we observe that BERT with intermediate task transfer learning fails to improve the performance of scientific keyphrase identification and classification potentially due to significant catastrophic forgetting. This result highlights that scientific knowledge achieved during the pre-training of language models on large scientific collections plays an important role in the target tasks. We also observe that sequence tagging related intermediate tasks, especially syntactic structure learning tasks such as POS Tagging, tend to work best for scientific keyphrase identification and classification.

show abstract

Section: Related Workmentioning

confidence: 99%