TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References

Abstract

In this paper, we introduce the semantic knowledge of medical images from their diagnostic reports to provide an inspirational network training and an interpretable prediction mechanism with our proposed novel multimodal neural network, namely TandemNet. Inside TandemNet, a language model is used to represent report text, which cooperates with the image model in a tandem scheme. We propose a novel dual-attention model that facilitates high-level interactions between visual and semantic information and effectively distills useful features for prediction. In the testing stage, TandemNet can make accurate image prediction with an optional report text input. It also interprets its prediction by producing attention on the image and text informative feature pieces, and further generating diagnostic report paragraphs. Based on a pathological bladder cancer images and their diagnostic reports (BCIDR) dataset, sufficient experiments demonstrate that our method effectively learns and integrates knowledge from multimodalities and obtains significantly improved performance than comparing baselines.

Deep learning in medical imaging and radiation therapy

“…In the testing stage, the input is a fundus image only, and the output is a probabilistic map of the lesion types in the image. Zhang et al . proposed a multimodal network that jointly learns from medical images and their diagnostic reports, in which semantic information interacts with visual information to improve the image understanding ability by teaching the network to distill informative features.…”

Computer-Aided Diagnostic System for Early Detection of Acute Renal Transplant Rejection Using Diffusion-Weighted MRI

“…In the testing stage, the input is a fundus image only, and the output is a probabilistic map of the lesion types in the image. Zhang et al 334 proposed a multimodal network that jointly learns from medical images and their diagnostic reports, in which semantic information interacts with visual information to improve the image understanding ability by teaching the network to distill informative features. Applied to bladder cancer images and the corresponding diagnostic reports, the network demonstrated improved performance compared to baseline CNN that only use image information for training.…”

Multi-Level Dual-Attention Based CNN for Macular Optical Coherence Tomography Classification

“…Attention mechanism has been explored for image captioning [17], voice activity detection [18], speech emotion recognition [19] and question answering [20]. For biomedical imaging, attention has been used for report generation [21], disease classification [22], [23], organ segmentation [24] and localization [25]. In [26], authors have introduced attention mechanism for macular OCT classification where the proposed deep network requires a large number of model parameters, but their performance evaluation is limited.…”