Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models

Abstract

Scientist learn early on how to cite scientific sources to support their claims. Sometimes, however, scientists have challenges determining where a citation should be situated-or, even worse, fail to cite a source altogether. Automatically detecting sentences that need a citation (i.e., citation worthiness) could solve both of these issues, leading to more robust and well-constructed scientific arguments. Previous researchers have applied machine learning to this task but have used small datasets and models that do not take advantage of recent algorithmic developments such as attention mechanisms in deep learning. We hypothesize that we can develop significantly accurate deep learning architectures that learn from large supervised datasets constructed from open access publications. In this work, we propose a Bidirectional Long Short-Term Memory (BiLSTM) network with attention mechanism and contextual information to detect sentences that need citations. We also produce a new, large dataset (PMOA-CITE) based on PubMed Open Access Subset, which is orders of magnitude larger than previous datasets. Our experiments show that our architecture achieves state of the art performance on the standard ACL-ARC dataset (F 1 = 0.507) and exhibits high performance (F 1 = 0.856) on the new PMOA-CITE. Moreover, we show that it can transfer learning across these datasets. We further use interpretable models to illuminate how specific language is used to promote and inhibit citations. We discover that sections and surrounding sentences are crucial for our improved predictions. We further examined purported mispredictions of the model, and uncovered systematic human mistakes in citation behavior and source data. This opens the door for our model to check documents during pre-submission and pre-archival procedures. We discuss limitations of our work and make this new dataset, the code, and a web-based tool available to the community. Modeling citation worthiness by using attention-based Bidirectional Long Short-Term Memory networks and interpretable modelsData sources and data pre-processing ACL Anthology Reference Corpus. The ACL Anthology Reference Corpus (ACL-ARC) is a collection of scientific articles in Computational Linguistics. The ACL-ARC 1.0 dataset consists of 10,921 articles up to February 2007, including the source PDF, automatically extracted full text, and the metadata for the articles. In order to use the ACL-ARC dataset, we need to remove some noisy sentences, such as footnotes, mathematical equations, and URLs. Bonab et al. (2018) carried out all these pre-processing steps and made the data available on the Internet 1 . This dataset consists of 85,778 sentences with citations and 1,142,275 sentences without citations. More statistics are presented in Table 2.PubMed Central Open Access Subset. PubMed Central Open Access subset (PMOAS) is a full-text collection of scientific literature in bio-medical and life sciences. PMOAS is created by the US's National Institutes of Health. We obtain a snapshot of PMOAS on August, 2...

On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles

“…Table 2 summarizes the results in terms of the precision, recall, F1 score for l c , and overall weighted F1 score. The baseline numbers reported here are either from prior works (Färber et al, 2018;Bonab et al, 2018;Zeng and Acuna, 2020) or based on architectures very similar to those used in these prior works. On the SEPID-cite dataset, our SC model obtained significantly better performance than the state-of-the-art results from (Zeng and The results on the ACL-cite dataset clearly show the importance of context in this domain.…”

“…This vector is then passed through a feed-forward layer to obtain the class label. This approach is similar to (Zeng and Acuna, 2020), where the authors used Glove embeddings (Pennington et al, 2014) to obtain sentence representations, and BiLSTMs for context representations. This formulation has also been used previously for question-answering (Devlin et al, 2019) and passage re-ranking (Nogueira and Cho, 2019).…”

On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles

“…This vector is then passed through a feed-forward layer to obtain the class label. This approach is similar to (Zeng and Acuna, 2020), where the authors used Glove embeddings (Pennington et al, 2014) to obtain sentence representations, and BiLSTMs for context representations. This formulation has also been used previously for question-answering (Devlin et al, 2019) and passage re-ranking (Nogueira and Cho, 2019).…”

“…Subsequent works from (Färber et al, 2018;Bonab et al, 2018) use similar approach but employ deep learning models like Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs). More recently, Zeng et al (Zeng and Acuna, 2020) proposed a Bidirectional Long shortterm memory (BiLSTM) based architecture and demonstrated that context, specifically the two adjacent sentences, can help improve the prediction of citation worthiness.…”

“…For the latter formulation, we propose a new hierarchical architecture, where the first layer provides sentence-level representations, and the second layer predicts the citation worthiness of the sentence sequence. We also introduce a new dataset mostly because the prior datasets (Bonab et al, 2018;Zeng and Acuna, 2020) do not have sufficient contextual information to study this research question.…”