Coreference annotation and resolution in the Colorado Richly Annotated Full Text (CRAFT) corpus of biomedical journal articles

Abstract

BackgroundCoreference resolution is the task of finding strings in text that have the same referent as other strings. Failures of coreference resolution are a common cause of false negatives in information extraction from the scientific literature. In order to better understand the nature of the phenomenon of coreference in biomedical publications and to increase performance on the task, we annotated the Colorado Richly Annotated Full Text (CRAFT) corpus with coreference relations.ResultsThe corpus was manually annotated with coreference relations, including identity and appositives for all coreferring base noun phrases. The OntoNotes annotation guidelines, with minor adaptations, were used. Interannotator agreement ranges from 0.480 (entity-based CEAF) to 0.858 (Class-B3), depending on the metric that is used to assess it. The resulting corpus adds nearly 30,000 annotations to the previous release of the CRAFT corpus. Differences from related projects include a much broader definition of markables, connection to extensive annotation of several domain-relevant semantic classes, and connection to complete syntactic annotation. Tool performance was benchmarked on the data. A publicly available out-of-the-box, general-domain coreference resolution system achieved an F-measure of 0.14 (B3), while a simple domain-adapted rule-based system achieved an F-measure of 0.42. An ensemble of the two reached F of 0.46. Following the IDENTITY chains in the data would add 106,263 additional named entities in the full 97-paper corpus, for an increase of 76% percent in the semantic classes of the eight ontologies that have been annotated in earlier versions of the CRAFT corpus.ConclusionsThe project produced a large data set for further investigation of coreference and coreference resolution in the scientific literature. The work raised issues in the phenomenon of reference in this domain and genre, and the paper proposes that many mentions that would be considered generic in the general domain are not generic in the biomedical domain due to their referents to specific classes in domain-specific ontologies. The comparison of the performance of a publicly available and well-understood coreference resolution system with a domain-adapted system produced results that are consistent with the notion that the requirements for successful coreference resolution in this genre are quite different from those of the general domain, and also suggest that the baseline performance difference is quite large.

Investigating Domain-Specific Information for Neural Coreference Resolution on Biomedical Texts

“…We employed two biomedical corpora: BioNLP Protein Coreference dataset (Nguyen et al, 2011) and CRAFT (Cohen et al, 2017). The BioNLP dataset consists of 1,210 PubMed abstracts selected from the GENIA-MedCo coreference corpus.…”

“…The BioNLP dataset consists of 1,210 PubMed abstracts selected from the GENIA-MedCo coreference corpus. CRAFT (Cohen et al, 2017) erence annotations of 67 full papers extracted from PMC. While BioNLP focusses on protein/gene coreference, CRAFT covers a wider range of coreference relations such as events, pronominal anaphora, noun phrases, verbs, and nominal premodifiers corefernce.…”

“…Prec Rec F1 (%) TEES (BioNLP ST) 67.2 14.4 23.8 ConcordU (BioNLP ST) 63.2 19.4 29.7 UZurich (BioNLP ST) 55.5 21.5 31.0 UUtah (BioNLP ST) 73.3 22.2 34.1 Choi et al (2016b) 46.3 50.0 48.1 PubMed+MM 55.6 47.5 51.2 Nguyen et al (2012) 50.2 52.5 51.3 62.7 50.4 55.9 D'Souza and Ng (2012) 55.6 67.2 60.9 Cohen et al (2017). This is not a fair comparison as our system only addressed identity chains and the test set is different from theirs.…”

“…In the biomedical domain, coreference information has been shown to enhance the performance of entity and event extraction Choi et al, 2016a). Most of work in this domain use rule-based or hybrid approaches (Nguyen et al, 2011(Nguyen et al, , 2012D'Souza and Ng, 2012;Li et al, 2014;Choi et al, 2016b;Cohen et al, 2017). These systems rely on syntactic parsers to extract hand-crafted features and rules, e.g., rules based on predicate argument structure (Nguyen et al, 2012; or features based on syntax trees (D'Souza and Ng, 2012).…”

Coreference Resolution in Full Text Articles with BERT and Syntax-based Mention Filtering

“…Coreference resolution is important not only in general domains but also in the biomedical domain. The Colorado Richly Annotated Full Text (CRAFT) corpus (Cohen et al, 2017) was constructed with an aim of boosting the performance of the task in the biomedical literature. Unlike other corpora, CRAFT is comprised of full text articles or full papers, its coreferent chains are arbitrarily long; the mean length of coreferent chains is 4 while the longest chain is 186, which makes the resolution even more difficult than usual.…”