Automated Mining of Leaderboards for Empirical AI Research

Abstract

We present a large-scale empirical investigation of the zero-shot learning phenomena in a specific recognizing textual entailment (RTE) task category, i.e. the automated mining of leaderboards for Empirical AI Research. The prior reported state-of-the-art models for leaderboards extraction formulated as an RTE task, in a non-zero-shot setting, are promising with above 90% reported performances. However, a central research question remains unexamined: did the models actually learn entailment? Thus, for the experiments in this paper, two prior reported state-of-the-art models are tested out-of-the-box for their ability to generalize or their capacity for entailment, given leaderboard labels that were unseen during training. We hypothesize that if the models learned entailment, their zero-shot performances can be expected to be moderately high as well-perhaps, concretely, better than chance. As a result of this work, a zero-shot labeled dataset is created via distant labeling formulating the leaderboard extraction RTE task.

Computer Science Named Entity Recognition in the Open Research Knowledge Graph

“…The longer ones have sentences from full-text articles, viz. ScienceIE [4], NLP-TDMS [34], SciREX [36], and ORKG-TDM [39]. We see that the corpora have had from one (e.g., the NCG corpus [23]) to atmost seven entity types (e.g., ACL-RD-TEC [59]).…”

“…E.g., the focus, technique, and domain entity types in the FTD corpus [31] helped examine the influence between research communities; ACL-RD-TEC [59] made possible a broader trends analysis with seven types. Eventually, corpora began to shed light on a novel scientific community research direction toward representing the entities as knowledge graphs [3] with hierarchical relation annotations such as synonymy [4] or semantic relations such 'Method Used-for a Task ' [44]; otherwise, scientific types were combined within full-fledged semantic constructs as Leaderboards with between three to four concepts [34,36,54,39], viz. research problem, dataset, method, metric, and score; or were in extraction objectives with solely contribution-centric entities of a paper [26,22].…”

“…Overall, the corpora served two main information extraction aims. 1) Some resources offered all mentions of CS relevant scientific entities: ACL-RD-TEC [59], ScienceIE [4], and STEM-ECR [24]; and 2) While others offered only contribution-centric entities: FTD [31], SciERC [44], NLP-TDMS [34], SciREX [36], NCG [23], ORKG-TDM [39], and CL-Titles [22].…”

“…Furthermore, since, in the context of this work, CS NER is qualified as being contribution-centric, only those entities that are either the outcome of a particular work or used to support the outcome of that work are candidate extraction targets. Related to the model we propose is the Task, Dataset, Metric (TDM) model attemped in several works [39,53,34] originally inspired from https://paperswithcode.com/. However, the TDM model targets only the automated construction of Leaderboards in the domain of Artificial Intelligence.…”

“…This mature field is also rife with various tools such as GATE (General Architecture for Text Engineering) [17], Mallet (Machine learning for language toolkit) [49], Natural Language Toolkit (Suite of Python libraries for NLP) [7], DBpedia Spotlight [51], Stanford NER [27], and SpaCy [33]. In contrast, CS NER has only recently started gaining traction following being organized as a task in a SemEval series having then garnered interest in the broader community [4,28,44,34,19,36,23,39,22].…”