Data preparation and interannotator agreement: BioCreAtIvE Task 1B

Abstract

Background: We prepared and evaluated training and test materials for an assessment of text mining methods in molecular biology. The goal of the assessment was to evaluate the ability of automated systems to generate a list of unique gene identifiers from PubMed abstracts for the three model organisms Fly, Mouse, and Yeast. This paper describes the preparation and evaluation of answer keys for training and testing. These consisted of lists of normalized gene names found in the abstracts, generated by adapting the gene list for the full journal articles found in the model organism databases. For the training dataset, the gene list was pruned automatically to remove gene names not found in the abstract; for the testing dataset, it was further refined by manual annotation by annotators provided with guidelines. A critical step in interpreting the results of an assessment is to evaluate the quality of the data preparation. We did this by careful assessment of interannotator agreement and the use of answer pooling of participant results to improve the quality of the final testing dataset.

Overview of BioCreAtIvE: critical assessment of information extraction for biology

“…This meant that we had to edit the gene lists to make them correspond to genes mentioned in the abstract, rather than all the genes curated in the full text article. We developed a procedure to automatically remove genes not found in the abstract and were able to provide a large quantity of "noisy" training data for the three organisms, together with small collections of carefully corrected development and test data [11]. We estimated the quality of the noisy training data for the three organisms.…”

“…There are 6 papers for task 1b, including an overview [10], an article describing preparation of the test sets and inter-annotator agreement experiments [11], and four articles describing systems and results for task 1b [9,12-14]. …”

A Methodology towards Effective and Efficient Manual Document Annotation: Addressing Annotator Discrepancy and Annotation Quality

“…Whenever new annotators joined the project, they had to be trained using previously annotated examples and follow the guideline. Colosimo et al [5] and Tanabe et al [28] also conduct corpus annotation in the biology domain and conclude that clear annotation guidelines are important, and the annotations should be validated by proper interannotator-agreement experiments.…”

Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems

“…3.6. Ideally, each topic would be judged by at least three judges and use a consensus-driven process which would provide maximum consistency throughout the entire process of gold standard creation (Hripcsak and Wilcox 2002;Colosimo et al 2005). The time scale of TREC and the resources required for a consensus approach made this impractical.…”

“…For the 2002 KDD cup and year 1 of the Genomics track, assessment was performed by using existing curation in FlyBase and Entrez Gene, respectively (Hersh and Bhupatiraju 2003;Yeh et al 2003). Task 1a of the first BioCreAtIvE challenge took a multipronged approach by developing guidelines for their three assessors, performing three-way comparisons of replicated judgments, and using pooled results from participants to uncover potential false negatives and false positives (Colosimo et al 2005). For judging duties that went beyond relevance assessment, the TREC Genomics track employed guidelines, training sessions, scaled-down ''mini-topics'', and moderated assessment by an experienced former judge.…”