Sentence, Phrase, and Triple Annotations to Build a Knowledge Graph of Natural Language Processing Contributions—A Trial Dataset

Abstract

Purpose This work aims to normalize the NlpContributions scheme (henceforward, NlpContributionGraph) to structure, directly from article sentences, the contributions information in Natural Language Processing (NLP) scholarly articles via a two-stage annotation methodology: 1) pilot stage—to define the scheme (described in prior work); and 2) adjudication stage—to normalize the graphing model (the focus of this paper). Design/methodology/approach We re-annotate, a second time, the contributions-pertinent information across 50 prior-annotated NLP scholarly articles in terms of a data pipeline comprising: contribution-centered sentences, phrases, and triple statements. To this end, specifically, care was taken in the adjudication annotation stage to reduce annotation noise while formulating the guidelines for our proposed novel NLP contributions structuring and graphing scheme. Findings The application of NlpContributionGraph on the 50 articles resulted finally in a dataset of 900 contribution-focused sentences, 4,702 contribution-information-centered phrases, and 2,980 surface-structured triples. The intra-annotation agreement between the first and second stages, in terms of F1-score, was 67.92% for sentences, 41.82% for phrases, and 22.31% for triple statements indicating that with increased granularity of the information, the annotation decision variance is greater. Research limitations NlpContributionGraph has limited scope for structuring scholarly contributions compared with STEM (Science, Technology, Engineering, and Medicine) scholarly knowledge at large. Further, the annotation scheme in this work is designed by only an intra-annotator consensus—a single annotator first annotated the data to propose the initial scheme, following which, the same annotator reannotated the data to normalize the annotations in an adjudication stage. However, the expected goal of this work is to achieve a standardized retrospective model of capturing NLP contributions from scholarly articles. This would entail a larger initiative of enlisting multiple annotators to accommodate different worldviews into a “single” set of structures and relationships as the final scheme. Given that the initial scheme is first proposed and the complexity of the annotation task in the realistic timeframe, our intra-annotation procedure is well-suited. Nevertheless, the model proposed in this work is presently limited since it does not incorporate multiple annotator worldviews. This is planned as future work to produce a robust model. Practical implications We demonstrate NlpContributionGraph data integrated into the Open Research Knowledge Graph (ORKG), a next-generation KG-based digital library with intelligent computations enabled over structured scholarly knowledge, as a viable aid to assist researchers in their day-to-day tasks. Originality/value NlpContributionGraph is a novel scheme to annotate research contributions from NLP articles and integrate them in a knowledge graph, which to the best of our knowledge does not exist in the community. Furthermore, our quantitative evaluations over the two-stage annotation tasks offer insights into task difficulty.

SciKGTeX - A LATEX Package to Semantically Annotate Contributions in Scientific Publications

“…Knowledgeable individuals annotate the triples for the ORKG via a web interface. Automatic extraction techniques to populate the knowledge graph are also used to some extent but are still in development phase [37]. Since the release of the ORKG web interface in the beginning of 2020, over 300 contributors added research contributions from publications to the knowledge graph.…”

Overview of STEM Science as Process, Method, Material, and Data Named Entities

“…Knowledge Graphs (KG) play a crucial role in many modern applications (https: //developers.google.com/knowledge-graph, accessed on 12 December 2022) as solutions to the information access and search problem. There have been several initiatives in the NLP [18,25,60,61], and the Semantic Web [62,63], communities suggesting an increasing trend toward adoption of KGs for scientific articles. The automatic construction of KGs from text is a challenging problem, more so owing to the multidisciplinary nature of Science at large.…”

Automated Mining of Leaderboards for Empirical AI Research

“…In other text mining initiatives, comprehensive knowledge mining themes are being defined on scholarly investigations. The recent NLPContributionGraph Shared Task [19,17,16] released KG annotations of contributions including the facets of research problem, approach, experimental settings, and results, in an evaluation series that showed it a challenging task. Similarly, in the Life Sciences, comprehensive KGs from reports of biological assays, wet lab protocols and inorganic materials synthesis reactions and procedures [7,8,31,32,33,41] are released as ontologized machine-interpretable formats for training machine readers.…”