SeqScore: Addressing Barriers to Reproducible Named Entity Recognition Evaluation

Abstract

To address a looming crisis of unreproducible evaluation for named entity recognition, we propose guidelines and introduce SeqScore, a software package to improve reproducibility. The guidelines we propose are extremely simple and center around transparency regarding how chunks are encoded and scored. We demonstrate that despite the apparent simplicity of NER evaluation, unreported differences in the scoring procedure can result in changes to scores that are both of noticeable magnitude and statistically significant. We describe SeqScore, which addresses many of the issues that cause replication failures.

Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023)

“…This section describes the features of SeqScore, focusing on the newest features that enable it to assist in many NER data workflows. Previous work (Palen-Michel et al, 2021) has described the scoring features of SeqScore, so they are not discussed in detail in this paper. SeqScore is released via PyPI (https://pypi.org/project/seqscore/) and development occurs on GitHub (https:// github.com/bltlab/seqscore).…”

“…Seq-Score supports several options to work with a wide variety of data files: setting the file encoding (older files often use ISO-8859-1), ignoring comment lines (which some files use for sentence provenance information), and automatic detection of field delimiters (older files use space, newer ones use tabs). Different strategies can be set regarding how to deal with invalid label transitions like O I-PER in BIO (for more details see Palen-Michel et al, 2021). SeqScore can maintain or discard the document boundaries specified using -DOCSTART-sentences inside CoNLL-format files, which enables scoring a reference with document boundaries against system output without them.…”

“…This paper describes the SeqScore toolkit and its applications for validating, summarizing, and transforming NER data. A previous publication (Palen-Michel et al, 2021), introduced SeqScore and described its value as a reproducibility-focused NER scorer. While this paper is also about Seq-Score, it has a different focus.…”

Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modeling

“…Evaluation for all models required extracted spans to match the annotation exactly in span and type to be correct. Evaluation was performed with SeqScore (Palen-Michel et al, 2021), using conlleval-style repair for invalid label sequences. All models were trained using an AMD 2990WX CPU and a single RTX 2080 Ti GPU.…”