Multilevel Text Alignment with Cross-Document Attention

Abstract

Text alignment finds application in tasks such as citation recommendation and plagiarism detection. Existing alignment methods operate at a single, predefined level and cannot learn to align texts at, for example, sentence and document levels. We propose a new learning approach that equips previously established hierarchical attention encoders for representing documents with a cross-document attention component, enabling structural comparisons across different levels (document-to-document and sentence-to-document). Our component is weakly supervised from document pairs and can align at multiple levels. Our evaluation on predicting document-to-document relationships and sentence-to-document relationships on the tasks of citation recommendation and plagiarism detection shows that our approach outperforms previously established hierarchical, attention encoders based on recurrent and transformer contextualization that are unaware of structural correspondence between documents.

Recovering Lexically and Semantically Reused Texts

“…Despite these factors, BERT-HAN's large performance drop on PAN is still surprising. However, we emphasize that even when using Zhou et al (2020)'s original numbers, BERT-HAN still lags behind both our lexical overlap baselines and fine-tuned BERT models, so our overall takeaways from §7 still stand. For the S2D task, our results are not directly comparable to the original numbers of Zhou et al (2020) for two reasons: positively-labeled target sentences.…”

“…However, we emphasize that even when using Zhou et al (2020)'s original numbers, BERT-HAN still lags behind both our lexical overlap baselines and fine-tuned BERT models, so our overall takeaways from §7 still stand. For the S2D task, our results are not directly comparable to the original numbers of Zhou et al (2020) for two reasons: positively-labeled target sentences. When there are fewer than k positively-labeled target sentences in an example, a perfect system will still have a P@k < 1.…”

“…However, as discussed in §1, we conduct a much larger set of experiments beyond those of Zhou et al (2020). In addition to the hierarchical neural models with document-level supervision proposed by Zhou et al (2020), we evaluate four sets of models: lexical overlap models, SoTA neural models trained for general paraphrase detection, hierarchical neural models with sentence-level supervision, and finetuned sequence-pair BERT models. Further, in Also related to our work is research studying sentence-pair problems, e.g.…”

“…GRU-HAN (deep) (Zhou et al, 2020): this model mirrors BERT-HAN, except with GloVe (Pennington et al, 2014) embeddings and a HAN with CDA at both the word and sentence level. It follows the same training and testing regime.…”

“…While many state-of-the-art (SoTA) NLP architectures have been trained on the closely-related tasks of document-and sentence-pair similarity detection (Reimers and Gurevych, 2019) and ad-hoc retrieval (Dai and Callan, 2019), prior methods for local text-reuse detection (LTRD) are mostly limited to lexical matching (Lee, 2007;Clough et al, 2002;Leskovec et al, 2009;Wilkerson et al, 2015;Smith et al, 2014) with some dictionary expansion (Moritz et al, 2016). To our knowledge, only Zhou et al (2020) has applied neural models to this problem, proposing hierarchical neural models that use a cross-document attention mechanism to model local similarities between two candidate documents.…”