Assisted annotation of medical free text using RapTAT

Gobbel, Glenn T.; Garvin, Jennifer H.; Reeves, Ruth; Cronin, Robert M.; Heavirland, Julia; Williams, Jenifer; Weaver, Allison; Jayaramaraja, Shrimalini; Giuse, Dario A.; Speroff, Theodore; Brown, Steven H.; Xu, Hua; Matheny, Michael E.

doi:10.1136/amiajnl-2013-002255

Cited by 38 publications

(25 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gobbel, et al [141] investigated the effect of pre-annotations on reducing the annotation time of clinical concepts compared to when annotating from scratch. As opposed to our proposed approach (i.e., to pre-annotate smartly selected samples by active learning models), they randomly selected a set of clinical notes to be preannotated using a ML model at each iteration.…”

Section: Background and Significancementioning

confidence: 99%

“…This is achieved by reducing the number of annotations that an annotator must add/modify/remove. Pre-annotated data, which is commonly generated using a dictionary [137] or existing NLP systems [138,139,140,141], have resulted in considerable less annotation time compared to fully manually annotated data. However, the accuracy of pre-annotations is directly correlated with savings in annotation time.…”

Section: Annotation Cost Analysis In Practicementioning

confidence: 99%

“…Although its constitution does not have a significant effect on the final AL annotation rate [176], a highly effective initial model learnt on the seed set can serve to provide high quality pre-annotations to annotators early in the AL process. Indeed, studies in fully supervised learning settings have shown that approaches based on high quality pre-annotations lead to lower manual annotation costs compared to an annotation-from-scratch approach [137,141,143].…”

Section: Analysis Of Seed Selection Strategiesmentioning

confidence: 99%

“…Pre-annotated data, which is commonly generated using a dictionary [137] or existing NLP systems [138,139,140,141,143], have resulted in significantly reduced annotation time compared to manually annotating the full dataset. Lingren, et al [137] used a dictionary-based approach to pre-annotate disease/disorder and sign/symptom entities with the aim of developing a gold standard for clinical named entity recognition in clinical trial announcements.…”

Section: Background and Significancementioning

confidence: 99%

“…They found that the time savings by pre-annotations are statistically higher than the de novo annotation. Gobbel, et al [141] were employed to do the annotation: (1) two annotators for manual annotations, and (2) the other two for reviewing the pre-annotated data generated by RapTAT. They found that interactive training using a human annotator in the loop increases the number of correct pre-annotations generated by RapTAT.…”

Section: Pre-annotation Settingsmentioning

confidence: 99%

See 4 more Smart Citations

Active Learning for Concept Extraction from Clinical Free Text

Kholghi¹

View full text Add to dashboard Cite

An increasing volume of clinical free-text data, such as discharge summaries and progress reports, has been collected by hospitals and healthcare centres and stored electronically for further processing. Extracting structured clinical information from such unstructured text resources is necessary for enabling secondary usage of reports, such as reporting, reasoning and retrieving, and for further processing in down-stream eHealth workflows. However, this analysis cannot be done manually, due to the high cost incurred by qualified experts to annotate the clinical free text. A significant initial step in extracting information from clinical free text is concept extraction, which involves identifying entities of interest in the clinical domain (such as diseases, medications, and symptoms). Currently, supervised machine learning approaches effectively extract clinical concepts by building powerful statistical models. However, these approaches require a large amount of high quality, annotated train data, which is created manually by domain experts through a costly and timeconsuming process. This results in a robust active learning framework for extracting clinical concepts, using state-of-the-art, active learning approaches. The second step is to leverage clinical information resources (i.e., terminologies and clinical information extraction tools) and other machine learning approaches (i.e., semi-supervised learning, unsupervised learning, and representation learning) to develop domain-specific and generic active learning approaches. This leads to a number of novel, active learning query strategies and a seed selection approach that outperform the state-of-the-art approaches with less manual annotation effort. The last step is to validate the benefits of the developed AL-based framework in reducing the annotation cost (i.e., time) through a comprehensive user study. An AL-assisted pre-annotation scheme is also introduced, in which the learning models built across the AL process generate high quality pre-annotations to be reviewed by human annotators. This further accelerates the annotation process, by significantly reducing the number of manual annotations that must be added or corrected compared to de novo annotation.The results of this study demonstrate that AL plays an important role in reducing the manual annotation cost. The CEAL framework extracts high quality domain concepts from clinical narratives, while significantly reducing the labour cost with up to 35% less annotation time required. Additionally, AL-assisted preannotations accelerate the de novo annotation process with a further 20% less annotation time required. This thesis contributes to information extraction from clinical unstructured text resources by alleviating the burden of manual annotation.The practical significance of this research is three-fold: (1) benefitting the overall patient healthcare by facilitating downstream eHealth workflows such as supporting clinical information processing, reporting, reasoning, and efficient decision m...

show abstract

Section: Background and Significancementioning

confidence: 99%

Section: Annotation Cost Analysis In Practicementioning

confidence: 99%

Section: Analysis Of Seed Selection Strategiesmentioning

confidence: 99%

Section: Background and Significancementioning

confidence: 99%

Section: Pre-annotation Settingsmentioning

confidence: 99%

See 3 more Smart Citations

Active Learning for Concept Extraction from Clinical Free Text

Kholghi¹

View full text Add to dashboard Cite

show abstract

Clinical information extraction using small data: An active learning approach based on sequence representations and word embeddings

Kholghi

Vine

Sitbon

et al. 2017

Asso for Info Science & Tech

View full text Add to dashboard Cite

This article demonstrates the benefits of using sequence representations based on word embeddings to inform the seed selection and sample selection processes in an active learning pipeline for clinical information extraction. Seed selection refers to choosing an initial sample set to label to form an initial learning model. Sample selection refers to selecting informative samples to update the model at each iteration of the active learning process. Compared to supervised machine learning approaches, active learning offers the opportunity to build statistical classifiers with a reduced amount of training samples that require manual annotation. Reducing the manual annotation effort can support automating the clinical information extraction process. This is particularly beneficial in the clinical domain, where manual annotation is a time-consuming and costly task, as it requires extensive labor from clinical experts. Our empirical findings demonstrate that (a) using sequence representations along with the length of sequence for seed selection shows potential towards more effective initial models, and (b) using sequence representations for sample selection leads to significantly lower manual annotation efforts, with up to 3% and 6% fewer tokens and concepts requiring annotation, respectively, compared to state-of-the-art query strategies.

show abstract