Known operator learning and hybrid machine learning in medical imaging—a review of the past, the present, and the future

Abstract

In this article, we perform a review of the state-of-the-art of hybrid machine learning in medical imaging. We start with a short summary of the general developments of the past in machine learning and how general and specialized approaches have been in competition in the past decades. A particular focus will be the theoretical and experimental evidence pro and contra hybrid modelling. Next, we inspect several new developments regarding hybrid machine learning with a particular focus on so-called known operator learning and how hybrid approaches gain more and more momentum across essentially all applications in medical imaging and medical image analysis. As we will point out by numerous examples, hybrid models are taking over in image reconstruction and analysis. Even domains such as physical simulation and scanner and acquisition design are being addressed using machine learning grey box modelling approaches. Towards the end of the article, we will investigate a few future directions and point out relevant areas in which hybrid modelling, meta learning, and other domains will likely be able to drive the state-of-the-art ahead.

Intrinsic Noise Improves Speech Recognition in a Computational Model of the Auditory Pathway

“…Our study provides evidence that an interplay of deep learning and neuroscience helps on the one hand to raise understanding of the function of biological neural networks and cognition in general (e.g., Schilling et al, 2018 , 2021b ; Krauss et al, 2019a , c , d , 2021 ; Gerum et al, 2020 ; Krauss and Maier, 2020 ; Bönsel et al, 2021 ; Metzner and Krauss, 2022 ), an emerging science strand referred to as cognitive computational neuroscience ( Kriegeskorte and Douglas, 2018 ). On the other hand, fundamental processing principles from nature—such as stochastic resonance—can be transferred to improve artificial neural systems, which is called neuroscience-inspired AI ( Hassabis et al, 2017 ; Gerum et al, 2020 ; Gerum and Schilling, 2021 ; Yang et al, 2021 ; Maier et al, 2022 ).…”

AutoSpeed: A Linked Autoencoder Approach for Pulse-Echo Speed-of-Sound Imaging for Medical Ultrasound

“…In this study, we proposed a novel approach for pulse-echo SoS imaging from a single plane-wave acquisition. Inspired by the known parameter paradigm [17,31], we hardcoded the domain transfer problem in a dual autoencoder approach, similar to the idea behind Cycle-GAN [32]. We trained two linked autoencoders to extract efficient representations from RF data and SoS maps.…”

“…In a fine-tuning step, a fully connected layer mapped the representation from the RF data domain to the SoS domain. There is evidence that the functions learned by networks can be decomposed into modules and those modules can be used in other tasks even without any further training [31,33,34]. Thus, in the inference, an encoder-decoder-like architecture is proposed in which the encoder is detached from the RF data autoencoder and the decoder is detached from the SoS autoencoder (both fully trained) and two paths are connected via a trained fully connected layer (IRM-Layer).…”

Deep learning based decoding of local field potential events

“…Summing up, following the trend of integrating artificial intelligence and neuroscience [76][77][78][79][80][81][82][83], machine learning provides valuable tools to extract information from electrophysiological data [84][85][86]. As described above in most studies the data is averaged over many measurement trials to increase the signal to noise-ratio.…”