Calbc Silver Standard Corpus

Abstract

The CALBC initiative aims to provide a large-scale biomedical text corpus that contains semantic annotations for named entities of different kinds. The generation of this corpus requires that the annotations from different automatic annotation systems be harmonized. In the first phase, the annotation systems from five participants (EMBL-EBI, EMC Rotterdam, NLM, JULIE Lab Jena, and Linguamatics) were gathered. All annotations were delivered in a common annotation format that included concept identifiers in the boundary assignments and that enabled comparison and alignment of the results. During the harmonization phase, the results produced from those different systems were integrated in a single harmonized corpus ("silver standard" corpus) by applying a voting scheme. We give an overview of the processed data and the principles of harmonization--formal boundary reconciliation and semantic matching of named entities. Finally, all submissions of the participants were evaluated against that silver standard corpus. We found that species and disease annotations are better standardized amongst the partners than the annotations of genes and proteins. The raw corpus is now available for additional named entity annotations. Parts of it will be made available later on for a public challenge. We expect that we can improve corpus building activities both in terms of the numbers of named entity classes being covered, as well as the size of the corpus in terms of annotated documents.

Cloud–Based Evaluation Framework for Big Data

“…However, due to the huge amount of data in the test set, manual annotation of all of it is infeasible. Therefore, a silver corpus approach, such as the one used in the CALBC challenges 17 [16], will be adopted. This silver corpus is built based on voting on the participant submissions.…”

Exploiting semantic annotations for open information extraction: an experience in the biomedical domain

“…The interest in semantic annotation of biomedical entities is such that initiatives such as CALBC (Collaborative Annotation of a Large Biomedical Corpus) [15] have been set up with the goal of providing a silver standard corpus (SSC) of annotated biomedical entities to the community. These annotations are the result of an agreement between the participants annotations.…”

“…More specifically, for each pair of semantic types expressed in the UMLS Semantic Network, we have selected relation strings that are included in other gold standards (e.g., protein-protein, drug-drug, protein-disease interactions) along with those that appear frequently in our dataset under the corresponding signature. Afterwards, we have manually clustered synonymous strings into 249 groups 15 . As in [54], we use precision (P), recall (R ) and F-score (F1) to measure the overlap between the best matches between the system-generated clusters and the GS groups.…”

An analysis on the entity annotations in biological corpora

“…Despite the undeniable presence of wrong annotations and the absence of many others, previous works have demonstrated that these corpora can support development of semisupervised or distant supervised systems for named-entity 13 and relationship extraction 14 . As manual annotation or validation is not required in this case, such corpora tends to be much larger than the gold-standard ones.…”