On coreference resolution performance metrics

Abstract

The paper proposes a Constrained Entity-Alignment F-Measure (CEAF) for evaluating coreference resolution. The metric is computed by aligning reference and system entities (or coreference chains) with the constraint that a system (reference) entity is aligned with at most one reference (system) entity. We show that the best alignment is a maximum bipartite matching problem which can be solved by the Kuhn-Munkres algorithm. Comparative experiments are conducted to show that the widelyknown MUC F-measure has serious flaws in evaluating a coreference system. The proposed metric is also compared with the ACE-Value, the official evaluation metric in the Automatic Content Extraction (ACE) task, and we conclude that the proposed metric possesses some properties such as symmetry and better interpretability missing in the ACE-Value.

Coreference Resolution for the Basque Language with BART

“…In the first one automatically detected mentions are provided to the models and in the second one the mentions are gold. 4 The metrics used in our evaluations are MUC (Vilain et al, 1995), B 3 (Bagga and Baldwin, 1998), CEAF e (Luo, 2005), CEAF m (Luo, 2005), and BLANC (Recasens and Hovy, 2011). The scores have been calculated using the reference implementation of the CoNLL scorer (Pradhan et al, 2014).…”

Generating Referring Expressions in Context: The GREC Task Evaluation Challenges

“…But intrinsic human judgements are simply not consistent and reliable enough to provide an objective meta-evaluation tool. 23 Moreover, all they provide is an insight into what humans (think they) like, not what is best or most useful for them (the two can be two very different matters, as discussed in [4]). …”

“…There does not appear to be a single standard evaluation metric in the coreference resolution community. We opted to use the following three: muc-6 [38], ceaf [23], and b-cubed [1], which seem to be the most widely accepted metrics. All three metrics compute Recall, Precision and F-Scores on aligned gold-standard and resolver-tool coreference chains.…”

Fast Gaussian Pairwise Constrained Spectral Clustering

“…We re-use features that are commonly used for mention pair classification (see e.g., [23], [4]), including grammatical type and subtypes, string and substring matches, apposition and copula, distance (number of separating mentions/sentences/words), gender and number match, synonymy/hypernym and animacy (based on WordNet), family name (based on closed lists), named entity types, syntactic features and anaphoricity detection. Evaluation metrics The systems' outputs are evaluated using the three standard coreference resolution metrics: MUC [29], B 3 [2], and Entity-based CEAF (or CEAF e ) [20]. Following the convention used in CoNLL-2012, we report a global F1-score (henceforth, CoNLL score), which corresponds to an unweighted average of the MUC, B 3 and CEAF e F1 scores.…”