Taming Pre-trained Language Models with N-gram Representations for Low-Resource Domain Adaptation

Abstract

Large pre-trained models such as BERT are known to improve different downstream NLP tasks, even when such a model is trained on a generic domain. Moreover, recent studies have shown that when large domain-specific corpora are available, continued pre-training on domain-specific data can further improve the performance of in-domain tasks. However, this practice requires significant domainspecific data and computational resources which may not always be available. In this paper, we aim to adapt a generic pretrained model with a relatively small amount of domain-specific data. We demonstrate that by explicitly incorporating the multi-granularity information of unseen and domain-specific words via the adaptation of (word based) ngrams, the performance of a generic pretrained model can be greatly improved. Specifically, we introduce a Transformer-based Domainaware N-gram Adaptor, T-DNA, to effectively learn and incorporate the semantic representation of different combinations of words in the new domain. Experimental results illustrate the effectiveness of T-DNA on eight lowresource downstream tasks from four domains. We show that T-DNA is able to achieve significant improvements compared to existing methods on most tasks using limited data with lower computational costs. Moreover, further analyses demonstrate the importance and effectiveness of both unseen words and the information of different granularities. 1

An Outbreak of NCIP (2019-nCoV) Infection in China — Wuhan, Hubei Province, 2019−2020

“…Recently, Domain Adaptation (DA) has gained popularity in the machine learning field due to its demonstrated effectiveness in enhancing model prediction performance, especially when dealing with unlabeled test samples. This applicability extends to domains like computer vision [36,54,56], as well as natural language processing [10,29,50]. As shown in Fig.…”

Active Prompting with Chain-of-Thought for Large Language Models

“…According to the nature of prompt tokens, existing prompt tuning methods can be categorized into two types based on their nature: 1) discrete prompts (Wallace et al, 2019;Shin et al, 2020;Yuan et al, 2021;Haviv et al, 2021;Gao et al, 2021;Ben-David et al, 2021;Davison et al, 2019;Su et al, 2022;Diao et al, 2022) and continuous prompts (Zhong et al, 2021;Qin and Eisner, 2021;Hambardzumyan et al, 2021;Han et al, 2021;Li and Liang, 2021). Discrete prompts optimize a sequence of discrete tokens, while continuous prompts optimize a sequence of vectors similar to adapter-based tuning (Houlsby et al, 2019;Pfeiffer et al, 2020;Diao et al, 2020Diao et al, , 2021. Our research is highly relevant to exemplar-based in-context learning and discrete prompts research.…”

PTEKC: Pre-training with Event knowledge of ConceptNet for Cross-lingual Event Causality Identification

“…Knowledge can be injected into PLMs by pre-training or fine-tuning, each corresponding to a separate research direction. During pre-training, the knowledge carried by knowledge graphs [49,53,54], entities [49,55], n-grams [56], knowledge embedding [57], synonym and hyponym-hypernym relations in WordNet [19], word-supersense knowledge [58], and knowledge bases [59][60][61] can be injected into PLMs by feeding knowledge inputs and designing new objectives. Furthermore, in terms of event pretraining, Majewska et al investigated the potential of leveraging knowledge about semantic-syntactic behaviour of verbs to improve the capacity of large pretrained models to reason about events in diverse languages [28].…”