Understanding spatial language in radiology: Representation framework, annotation, and spatial relation extraction from chest X-ray reports using deep learning

Datta, Sushmita; Si, Yuqi; Rodriguez, Laritza; Shooshan, Sonya E.; Demner‐Fushman, Dina; Roberts, Kirk

doi:10.1016/j.jbi.2020.103473

Cited by 20 publications

(22 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The LSTM is an extension of the basic RNN. This network adds time-dependent features relying on a preceding timestamp and operates as memory cell for remembering data from the preceding timestamp [18,29]. The memory cell c is controlled through a group of gate networks (Figure 1b), including f forget gate network, i input gate network, and o output gate network.…”

Section: Long-short-term-memory Modelsmentioning

confidence: 99%

“…Recurrent Neural Network (RNN) models are leading methods to deeply learn the longitudinal data. A variant of RNN is long-shortterm-memory (LSTM) [18] that captures both long-term and short-term dependencies within sequential data. Gao et al [19] employed distanced LSTM with time-distanced gates for diagnosing lung cancer by using both real computed tomography images and simulated data.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Multi-Modal Evolutionary Deep Learning Model for Ovarian Cancer Diagnosis

et al. 2021

View full text Add to dashboard Cite

Ovarian cancer (OC) is a common reason for mortality among women. Deep learning has recently proven better performance in predicting OC stages and subtypes. However, most of the state-of-the-art deep learning models employ single modality data, which may afford low-level performance due to insufficient representation of important OC characteristics. Furthermore, these deep learning models still lack to the optimization of the model construction, which requires high computational cost to train and deploy them. In this work, a hybrid evolutionary deep learning model, using multi-modal data, is proposed. The established multi-modal fusion framework amalgamates gene modality alongside with histopathological image modality. Based on the different states and forms of each modality, we set up deep feature extraction network, respectively. This includes a predictive antlion-optimized long-short-term-memory model to process gene longitudinal data. Another predictive antlion-optimized convolutional neural network model is included to process histopathology images. The topology of each customized feature network is automatically set by the antlion optimization algorithm to make it realize better performance. After that the output from the two improved networks is fused based upon weighted linear aggregation. The deep fused features are finally used to predict OC stage. A number of assessment indicators was used to compare the proposed model to other nine multi-modal fusion models constructed using distinct evolutionary algorithms. This was conducted using a benchmark for OC and two benchmarks for breast and lung cancers. The results reveal that the proposed model is more precise and accurate in diagnosing OC and the other cancers.

show abstract

Section: Long-short-term-memory Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Multi-Modal Evolutionary Deep Learning Model for Ovarian Cancer Diagnosis

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Earlier machine learning systems employed tradition methods such as Logistic Regression [1,8], Decision Trees [25], Support Vector Machine (SVM) [8], Random Forests [8], Conditional Markov Model (CMM) as well as Conditional Random Field (CRF) [9]. Recent advancements in Deep Neural Networks (DNNs) lead to the exploration of Recurrent Neural Network (RNN) [5,21], Convolutional Neural Network (CNN) [17], as well as Transformers [5] such as BERT [7] and XLNet [26] for information extraction.…”

Section: Machine Learning Approachesmentioning

confidence: 99%

“…Radiology reports in the form of Electronic Health Report (EHR) can be a rich source of data for medical cohort studies, image labelling, and decision support after information extraction techniques are applied for automatic annotation and labeling. Despite abundant researches in recent years, existing information extraction systems of radiology reports are developed for either general [2,19] or Chest X-ray radiology reports [1,5,9,10,[12][13][14]20]. Although some of these system can capture the presence or absence of bone fracture, they either do not extract other significant observations [8,25], such as the uncertainty, type and location of bone fracture, or rely on regular expressions [22][23][24], which hamper their performance.…”

Section: Introductionmentioning

confidence: 99%

BoneBert: A BERT-based Automated Information Extraction System of Radiology Reports for Bone Fracture Detection and Diagnosis

Dai

Zhong²,

Han

2021

Advances in Intelligent Data Analysis XIX

View full text Add to dashboard Cite

Radiologists make the diagnoses of bone fractures through examining X-ray radiographs and document them in radiology reports. Applying information extraction techniques on such radiology reports to retrieve the information of bone fracture diagnosis could yield a source of structured data for medical cohort studies, image labelling and decision support concerning bone fractures. In this study, we proposed an information extraction system of Bone X-ray radiology reports to retrieve the details of bone fracture detection and diagnosis, based on a bio-medically pre-trained Bidirectional Encoder Representations from Transformers (BERT) natural language processing (NLP) model by Google. The model, named as BoneBert, was first trained on annotations automatically generated by a handcrafted rule-based labelling system using a dataset of 6,048 X-ray radiology reports and then finetuned on a small set of 4,890 expert annotations. Thus, the model was trained in a "semi-supervised" fashion. We evaluated the performance of the proposed model and compared it with the conventional rule-based labelling system on two typical tasks: Assertion Classification (AC) for bone fracture status detection (positive, negative or uncertainty) and Named Entity Recognition (NER) related to the fracture type, the bone type and location of a fracture occurs. BoneBert outperformed the rulebased system in both tasks, showing great potential for automated information extraction of the detection and diagnosis of bone fracture from radiology reports, such as, the clinical status, type and location of bone fracture, and more related observations.

show abstract

“…This annotated dataset can be used for developing automatic approaches targeted toward spatial information extraction from radiology reports which then can be applied to numerous clinical applications. We utilize this dataset to develop deep learning-based methods for automatically extracting the Spatial Indicator s as well as the associated spatial roles [1] .…”

mentioning

confidence: 99%

A dataset of chest X-ray reports annotated with Spatial Role Labeling annotations

Datta

Roberts

2020

Data in Brief

Self Cite

View full text Add to dashboard Cite

In this paper, we present a dataset consisting of 2000 chest X-ray reports (available as part of the Open-i image search platform) annotated with spatial information. The annotation is based on Spatial Role Labeling. The information includes annotating a radiographic finding, its associated anatomical location, any potential diagnosis described in connection to the spatial relation (between finding and location), and any hedging phrase used to describe the certainty level of a finding/diagnosis. All these annotations are identified with reference to a spatial expression (or Spatial Indicator ) that triggers a spatial relation in a sentence. The spatial roles used to encode the spatial information are Trajector , Landmark , Diagnosis , and Hedge . In total, there are 1962 Spatial Indicator s (mainly prepositions). There are 2293 Trajector s, 2167 Landmark s, 455 Diagnosis , and 388 Hedge s in the dataset. This annotated dataset can be used for developing automatic approaches targeted toward spatial information extraction from radiology reports which then can be applied to numerous clinical applications. We utilize this dataset to develop deep learning-based methods for automatically extracting the Spatial Indicator s as well as the associated spatial roles [1] .

show abstract

Understanding spatial language in radiology: Representation framework, annotation, and spatial relation extraction from chest X-ray reports using deep learning

Cited by 20 publications

References 35 publications

Multi-Modal Evolutionary Deep Learning Model for Ovarian Cancer Diagnosis

Multi-Modal Evolutionary Deep Learning Model for Ovarian Cancer Diagnosis

BoneBert: A BERT-based Automated Information Extraction System of Radiology Reports for Bone Fracture Detection and Diagnosis

A dataset of chest X-ray reports annotated with Spatial Role Labeling annotations

Contact Info

Product

Resources

About