Machine Learning vs. Rules and Out-of-the-Box vs. Retrained

Abstract

Bibliographic reference parsing refers to extracting machinereadable metadata, such as the names of the authors, the title, or journal name, from bibliographic reference strings. Many approaches to this problem have been proposed so far, including regular expressions, knowledge bases and supervised machine learning. Many open source reference parsers based on various algorithms are also available. In this paper, we apply, evaluate and compare ten reference parsing tools in a specific business use case. The tools are Anystyle-Parser, Biblio, CERMINE, Citation, Citation-Parser, GROBID, ParsCit, PDFSSA4MET, Reference Tagger and Science Parse, and we compare them in both their outof-the-box versions and versions tuned to the project-specific data. According to our evaluation, the best performing out-of-thebox tool is GROBID (F1 0.89), followed by CERMINE (F1 0.83) and ParsCit (F1 0.75). We also found that even though machine learning-based tools and tools based on rules or regular expressions achieve on average similar precision (0.77 for MLbased tools vs. 0.76 for non-ML-based tools), applying machine learning-based tools results in a recall three times higher than in the case of non-ML-based tools (0.66 vs. 0.22). Our study also confirms that tuning the models to the task-specific data results in the increase in the quality. The retrained versions of reference parsers are in all cases better than their out-of-the-box counterparts; for GROBID F1 increased by 3% (0.92 vs. 0.89), for CERMINE by 11% (0.92 vs. 0.83), and for ParsCit by 16% (0.87 vs. 0.75).

S2ORC: The Semantic Scholar Open Research Corpus

“…Ahmad and Afzal (2018) evaluate GROBID for de- tecting inline citations using a corpus of 5k Cite-Seer papers, and found GROBID to have an F1score of 0.89 on this task. Tkaczyk et al (2018) report GROBID as the best among 10 out-of-the-box tools for parsing bibliographies, also achieving an F1 of 0.89 in an evaluation corpus of 9.5k papers.…”

Geo-Quantities: A Framework for Automatic Extraction of Measurements and Spatial Context from Scientific Documents

“…Popular approaches include regular expressions, knowledge bases, supervised machine learning, and hybrid approaches. Regular expressions are usually combined with additional approaches, for example with knowledge bases such as thesauri or ontologies, however in such approaches the system must first be filled with available knowledge [13]. Recently in 2019, an unsupervised rule-based approach was proposed that identifies units in source data and provides a corresponding semantic representation based on NASA's QUDT (Quantity, Unit, Dimension and Type) ontology using Arpeggio as a grammar parser [11].…”

“…In a supervised machine learning-based approach, measurement parsing is usually formally defined as a sequence labelling problem encompassing a variety of tasks, e.g. part-of-speech (POS) tagging or named-entity recognition (NER) [13]. Most of the existing tools are trainable, which means that they are able to automatically learn complex features and adapt parsing rules from training data.…”

A flexible template generation and matching method with applications for publication reference metadata extraction

“…These RME methods have their merits and shortcomings, as presented in Tkaczyk, Collins, Sheridan, and Beel's (2018) comprehensive comparison of these methods. Although machine learning‐based approaches require minimal human involvement (aside from annotation on training data) to obtain satisfactory performance, they suffer from data sparseness and a lack of generality.…”