Multi-task SonoEyeNet: Detection of Fetal Standardized Planes Assisted by Generated Sonographer Attention Maps

Abstract

We present a novel multi-task convolutional neural network called Multi-task SonoEyeNet (M-SEN) that learns to generate clinically relevant visual attention maps using sonographer gaze tracking data on input ultrasound (US) video frames so as to assist standardized abdominal circumference (AC) plane detection. Our architecture consists of a generator and a discriminator, which are trained in an adversarial scheme. The generator learns sonographer attention on a given US video frame to predict the frame label (standardized AC plane / background). The discriminator further fine-tunes the predicted attention map by encouraging it to mimick the ground-truth sonographer attention map. The novel model expands the potential clinical usefulness of a previous model by eliminating the requirement of input gaze tracking data during inference without compromising its plane detection performance (Precision: 96.8, Recall: 96.2, F-1 score: 96.5).

Transforming obstetric ultrasound into data science using eye tracking, voice recording, transducer motion and ultrasound video

“…By measuring different sonographer actions or tasks, we further aim to understand better several topics such as how to effectively and efficiently initially train sonographers, monitor learning progress, and enhance scanning workflow. Our data can also help design deep learning applications for assistive automation technology as described elsewhere 33 – 36 .…”

“…Previous work in ultrasound learning and workflow in non-obstetric ultrasound fields has resulted in the creation of automated algorithms to aid transducer guidance, automatic scanning, and image analysis 37 , 38 . In obstetrics, real-time automatic identification of standard planes may aid diagnosis and quality assurance 33 , 36 , 39 – 46 .…”

Calibrated Bayesian Neural Networks to Estimate Gestational Age and Its Uncertainty on Fetal Brain Ultrasound Images

“…Each CB f,sz,st was parameterized by the shared filter size, kernel size and stride respectively. The VGG-16 like backbone, in this notation, is CB 64,3,2 -CB 128,3,2 -CB 256,3,2 -CB 512,3,2 -CB 512, 3,2 Epistemic and Aleatoric Uncertainties We can break down uncertainty estimates into uncertainty over the network weights (epistemic uncertainty) and irreducible uncertainties over the noise inherent in the data (aleatoric uncertainty) [11]. As training data size increases, in theory epistemic uncertainty converges to zero.…”

“…GA estimation based directly on image appearance would reduce the need for human interaction of automated biometry after image capture. Further, significant progress has been made in automatic plane finding [2,12], meaning that our algorithm might, in the future, be incorporated in a fully automated ultrasound-based GA estimation solution for minimally trained healthcare professional in a global health setting [23].…”

Semi-supervised Learning of Fetal Anatomy from Ultrasound

“…However, low image quality and shortage of experts can compromise screening efficacy or in the least make quality heterogeneous across sites. To democratize care, several efforts have been made to automate standard plane detection using deep learning [2,3,4,8]. However, many of these methods still rely on large amounts of labelled data.…”