PARADE: A New Dataset for Paraphrase Identification Requiring Computer Science Domain Knowledge

Abstract

We present a new benchmark dataset called PARADE for paraphrase identification that requires specialized domain knowledge. PA-RADE contains paraphrases that overlap very little at the lexical and syntactic level but are semantically equivalent based on computer science domain knowledge, as well as nonparaphrases that overlap greatly at the lexical and syntactic level but are not semantically equivalent based on this domain knowledge. Experiments show that both state-of-the-art neural models and non-expert human annotators have poor performance on PARADE. For example, BERT after fine-tuning achieves an F1 score of 0.709, which is much lower than its performance on other paraphrase identification datasets. PARADE can serve as a resource for researchers interested in testing models that incorporate domain knowledge. We make our data and code freely available. 1

Knowledge-Guided Paraphrase Identification

“…We use two public datasets, PARADE (He et al, 2020a) and clinical-STS2019 , to evaluate the model performance. PARADE is a computer science domain benchmark dataset for paraphrase identification, while clinicalSTS2019 belongs to the biomedical domain.…”

“…PARADE is a computer science domain benchmark dataset for paraphrase identification, while clinicalSTS2019 belongs to the biomedical domain. For PARADE dataset, we use the same training, validation, testing splits with He et al (2020a). In clinicalSTS2019, the similarity score of each sentence pair ranges from 0 to 5, where 0 indicates irrelevance, and 5 indicates the equivalence in semantic meanings between the two sentences.…”

“…In fact, each meaningful sentence usually belongs to a certain domain and contains domainspecific knowledge (He et al, 2020a). When domain experts identify whether these two sentences are paraphrases or not, they first read sentences to comprehend the semantic meanings, and then analyze them based on the domain knowledge associated with the sentences, and finally make a decision.…”

“…"IMP" represents the improvement brought by Knowing. The results of baselines on PARADE are reported fromHe et al (2020a).…”

Towards Domain-Generalizable Paraphrase Identification by Avoiding the Shortcut Learning

“…Neural network based models have been proposed for the supervised PI task, and achieve decent performance in the single-domain setting (Yin and Schütze, 2015;Wang et al, 2017;Yang et al, 2019). At present, the existing PI corpora are restricted to several particular domains (Dolan et al, 2004;Xu et al, 2014;He et al, 2020), while the practical sentence pair for the paraphrase judgment can be from any unlabeled domain. At the same time, building a PI corpus for a novel domain needs massive human effort and is expensive.…”