Understanding the meanings of citations using sentiment, role, and citation function classifications

Abstract

Traditional citation analyses use quantitative methods only, even though there is meaning in the sentences containing citations within the text. This article analyzes three citation meanings: sentiment, role, and function. We compare citation meanings patterns between fields of science and propose an appropriate deep learning model to classify the three meanings automatically at once. The data comes from Indonesian journal articles covering five different areas of science: food, energy, health, computer, and social science. The sentences in the article text were classified manually and used as training data for an automatic classification model. Several classic models were compared with the proposed multi-output convolutional neural network model. The manual classification revealed similar patterns in citation meaning across the science fields: (1) not many authors exhibit polarity when citing, (2) citations are still rarely used, and (3) citations are used mostly for introductions and establishing relations instead of for comparisons with and utilizing previous research. The proposed model’s automatic classification metric achieved a macro F1 score of 0.80 for citation sentiment, 0.84 for citation role, and 0.88 for citation function. The model can classify minority classes well concerning the unbalanced dataset. A machine model that can classify several citation meanings automatically is essential for analyzing big data of journal citations.

Adaptive sentiment analysis using multioutput classification: a performance comparison

“…After the data is available, the next step is to select the algorithm to be incorporated into the model ( Budi & Yaniasih, 2022 ; Jafarian et al, 2021 ; Kaur et al, 2021 ). In general, machine learning algorithms such as Naive Bayes, SVM, and neural networks are utilized in sentiment analysis.…”

Sentiment Dimensions and Intentions in Scientific Analysis: Multilevel Classification in Text and Citations

“…Sentiment analysis of citations has attracted particular attention for two main reasons: First, to improve bibliometric metrics by focusing primarily on the quality rather than quantity of citations, with the aim of reducing bias and providing evidence-based support for writing. Second, to detect non-reproducible research, i.e., the identification of research papers or results that cannot be replicated or verified by other researchers, especially in the biomedical field, where unfavorable attitudes may be early indicators of the non-reproducibility of research, thus saving time and resources [28]. Therefore, although positive polarity citations have a significant impact on science, as they can enhance the validity and reliability of findings and even promote the reputation and career of researchers, the study by Catalini et al [29], however, equally highlights that negative citations can also play an important role in science.…”

Multi-sentence and multi-intent classification using RoBERTa and graph convolutional neural network

“…Additionally, they performed intent classification on the SciCite dataset and achieved an 85% accuracy using XLNet. Budi & Yaniasih, conducted experiments on 9173 citation sentences from five science disciplines: Food, Energy, Health, Social, and Computer [6]. They employed a CNN-based multi-task learning approach for citation sentiment classification (positive, negative, and neutral), citation role classification (supplemental, result, method, and data), and citation function classification (introducing, relating, utilizing, explaining, and comparing).…”

“…This limited approach fails to provide a comprehensive understanding since there may be preceding or following sentences that discuss the same citing paper with different intentions. Consequently, a single-sentence context is insufficient for thorough citation analysis [6]. To overcome the limitation posed by a single citation sentence or a fixed window of citation sentences, we advocate to consider multi-sentence-based citation context along with its potential intents [7].…”