Using an ensemble system to improve concept extraction from clinical records

Abstract

Recognition of medical concepts is a basic step in information extraction from clinical records. We wished to improve on the performance of a variety of concept recognition systems by combining their individual results. We selected two dictionary-based systems and five statistical-based systems that were trained to annotate medical problems, tests, and treatments in clinical records. Manually annotated clinical records for training and testing were made available through the 2010 i2b2/VA (Informatics for Integrating Biology and the Bedside) challenge. Results of individual systems were combined by a simple voting scheme. The statistical systems were trained on a set of 349 records. Performance (precision, recall, F-score) was assessed on a test set of 477 records, using varying voting thresholds. The combined annotation system achieved a best F-score of 82.2% (recall 81.2%, precision 83.3%) on the test set, a score that ranks third among 22 participants in the i2b2/VA concept annotation task. The ensemble system had better precision and recall than any of the individual systems, yielding an F-score that is 4.6% point higher than the best single system. Changing the voting threshold offered a simple way to obtain a system with high precision (and moderate recall) or one with high recall (and moderate precision). The ensemble-based approach is straightforward and allows the balancing of precision versus recall of the combined system. The ensemble system is freely available and can easily be extended, integrated in other systems, and retrained.

Concept selection for phenotypes and diseases using learn to rank

“…Ensembles have though been used before for the recognition of clinical concepts. Kang et al [ 36 ] for example employed dictionary and statistical pattern based techniques on the 2010 I2B2 corpus of EPRs, for term recognition (but not concept normalisation) achieving the third level of performance in the shared task. Xia et al [ 37 ] show the effects of combining MetaMap and cTAKES for the same ShARE/CLEF data we have shown here.…”

Stacked Generalization for Medical Concept Extraction from Clinical Notes

“…Both architectures are described below. Voting Ensemble Method: We implemented the majority voting strategy suggested by Kang et al (2012) with a simple modification to avoid labeling concepts with overlapping text spans. When two different concepts have overlapping text spans, the concept that receives more votes is selected.…”

Biomedical literature classification using encyclopedic knowledge: a Wikipedia-based bag-of-concepts approach

“…Jonquet, Shah & Musen (2009) present Open Biomedical Annotator: first, it extracts terms from text documents making use of Mgrep ( Dai et al, 2008 ); second, it maps these terms to biomedical concepts from UMLS and other biomedical ontologies from the National Centre from Biomedical Ontologies (NCBO); and, finally, it annotates the documents with these concepts. Kang et al (2012) combine seven domain-specific annotators—ABNER, Lingpipe, MetaMap, OpenNLP Chunker, JNET, Peregrine and StandforNer—to extract medical concepts from clinical texts, providing better results than any of the individual systems alone. Several authors make use of these and other semantic annotators for biomedical classification tasks such as: Yetisgen-Yildiz & Pratt (2005) , who use MetaMap to extract concepts from documents and use it to classify biomedical literature; and Zhou, Zhang & Hu (2008a) , who use a semantic annotator based on UMLS (MaxMatcher ( Zhou, Zhang & Hu, 2006 )) for the Bayesian classification of the biomedical literature corpus OHSUMED.…”