Observational Supervision for Medical Image Classification Using Gaze Data

Saab, Khaled; Hooper, Sarah M.; Sohoni, Nimit S.; Parmar, Jupinder; Pogatchnik, Brian; Wu, Sen; Dunnmon, Jared; Zhang, Hongyang R.; Rubin, Daniel L.; Ré, Christopher

doi:10.1007/978-3-030-87196-3_56

Cited by 16 publications

(14 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conduct the experiments on two datasets: INbreast [34] and SIIM-ACR [44,50]. The INbreast dataset [34] includes 410 full-field digital mammography images which were collected during lowdose X-ray irradiation of the breast.…”

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

“…According to BI-RADS [24] assessment of masses, the images can be classified into three groups: normal(302), benign (37) and malignant(71), respectively. Saab et al [44] randomly selected 1,170 images, with 268 cases of pneumon, from the SIIM-ACR Pneumothorax dataset [50] and collected gaze data from three radiologists. We randomly split the dataset into 80% and 20% as training and testing dataset.…”

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

See 1 more Smart Citation

Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Ma¹,

Zhang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning harmful shortcuts such as spurious correlations and biases prevents deep neural networks from learning the meaningful and useful representations, thus jeopardizing the generalizability and interpretability of the learned representation. The situation becomes even more serious in medical imaging, where the clinical data (e.g., MR images with pathology) are limited and scarce while the reliability, generalizability and transparency of the learned model are highly required. To address this problem, we propose to infuse human experts' intelligence and domain knowledge into the training of deep neural networks. The core idea is that we infuse the visual attention information from expert radiologists to proactively guide the deep model to focus on regions with potential pathology and avoid being trapped in learning harmful shortcuts. To do so, we propose a novel eye-gaze-guided vision transformer (EG-ViT) for diagnosis with limited medical image data. We mask the input image patches that are out of the radiologists' interest and add an additional residual connection in the last encoder layer of EG-ViT to maintain the correlations of all patches. The experiments on two public datasets of INbreast and SIIM-ACR demonstrate our EG-ViT model can effectively learn/transfer experts' domain knowledge and achieve much better performance than baselines. Meanwhile, it successfully rectifies the harmful shortcut learning and significantly improves the EG-ViT model's interpretability. In general, EG-ViT takes the advantages of both human expert's prior knowledge and the power of deep neural networks. This work opens new avenues for advancing current artificial intelligence paradigms by infusing human intelligence.

show abstract

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Ma¹,

Zhang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…CAT2000 [3] has 2000 training images and 2000 testing images from 20 different categories with eye tracking data. The SIIM-ACR [49] is a chest X-ray dataset with only pneumothorax disease, and recently work [46] randomly selected 1,170 images, with 268 cases of pneumothorax and collected gaze data from three radiologists. Follow this work [46], we chose these 1170 images and randomly split them into 80% and 20% as training and testing dataset.…”

Section: Datasets and Evaluation Metricsmentioning

confidence: 99%

Rectify ViT Shortcut Learning by Visual Saliency

Ma¹,

Zhang²,

Chen³

et al. 2022

Preprint

View full text Add to dashboard Cite

Shortcut learning is common but harmful to deep learning models, leading to degenerated feature representations and consequently jeopardizing the model's generalizability and interpretability. However, shortcut learning in the widely used Vision Transformer (ViT) framework is largely unknown. Meanwhile, introducing domain-specific knowledge is a major approach to rectifying the shortcuts, which are predominated by background related factors. For example, in the medical imaging field, eye-gaze data from radiologists is an effective human visual prior knowledge that has the great potential to guide the deep learning models to focus on meaningful foreground regions of interest. However, obtaining eye-gaze data is time-consuming, labor-intensive and sometimes even not practical. In this work, we propose a novel and effective saliency-guided vision transformer (SGT) model to rectify shortcut learning in ViT with the absence of eye-gaze data. Specifically, a computational visual saliency model (either pre-trained or fine-tuned) is adopted to predict saliency maps for input image samples. Then, the saliency maps are used to distil the most informative image patches. In the proposed SGT, the self-attention Preprint. Under review.

show abstract

“…In parallel to our project, Karargyris et al built a dataset of eye-tracking data and respective dictation of CXR reports, containing 1,083 readings by only one radiologist 18 . Saab et al built a dataset of eye-tracking data for the task of identifying pneumothoraces for 1,170 CXRs 19 . Still, their dataset does not contain dictations and focuses on the general differences in eye-tracking data for normal and abnormal images.…”

Section: Background and Summarymentioning

confidence: 99%

REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays

Lanfredi,

Zhang,

Auffermann

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning has shown recent success in classifying anomalies in chest x-rays, but datasets are still small compared to natural image datasets. Supervision of abnormality localization has been shown to improve trained models, partially compensating for dataset sizes. However, explicitly labeling these anomalies requires an expert and is very time-consuming. We propose a method for collecting implicit localization data using an eye tracker to capture gaze locations and a microphone to capture a dictation of a report, imitating the setup of a reading room, and potentially scalable for large datasets. The resulting REFLACX (Reports and Eye-Tracking Data for Localization of Abnormalities in Chest X-rays) dataset was labeled by five radiologists and contains 3,032 synchronized sets of eye-tracking data and timestamped report transcriptions. We also provide bounding boxes around lungs and heart and validation labels consisting of ellipses localizing abnormalities and image-level labels. Furthermore, a small subset of the data contains readings from all radiologists, allowing for the calculation of inter-rater scores.

show abstract

Observational Supervision for Medical Image Classification Using Gaze Data

Cited by 16 publications

References 30 publications

Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Eye-gaze-guided Vision Transformer for Rectifying Shortcut Learning

Rectify ViT Shortcut Learning by Visual Saliency

REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays

Contact Info

Product

Resources

About