Live Monitoring of Haemodynamic Changes with Multispectral Image Analysis

Abstract

State-of-the-art concepts in the field of computer assisted medical interventions are typically based on registering pre-operative imaging data to the patient. While this approach has many relevant clinical applications, it suffers from one core bottleneck: it cannot account for tissue dynamics because it works with “offline” data. To overcome this issue, we propose a new approach to surgical imaging that combines the power of multispectral imaging with the speed and robustness of deep learning based image analysis. Core innovation is an end-to-end deep learning architecture that integrates all preprocessing steps as well as the actual regression task in a single network. According to a quantitative in silico validation, our approach is well-suited for solving the inverse problem of relating multispectral image pixels to underlying functional tissue properties in real time. A porcine study further suggests that our method is capable of monitoring haemodynamic changes in vivo. Deep learning based multispectral imaging could thus become a valuable tool for imaging tissue dynamics.

Spectral organ fingerprints for intraoperative tissue classification with hyperspectral imaging

“…Over the last few years, there have been extensive efforts to implement HSI technology in healthcare. Examples of potential future clinical applications comprise the objective evaluation of tissue oxygenation and blood perfusion [3-6], inflammation and sepsis [7] or malignancy [8] as well as computer-assisted decision-making and automated organ identification [9]. These have the potential to support future developments such as intraoperative cognitive assistance systems or even automatization of robotic surgery.…”

Surgical spectral imaging

“…(2015) Spectral Filter wheel 420–972 10 8 1392 × 1040 - (not given) Open Skin Wirkert et al. (2016) ; Ayala et al. (2019) Spectral Filter wheel 470–700 20–25 8 1228 × 1029 2.5 (0.18 s) Laparoscopy, open Bowel, brain Olweny et al.…”

“…Reported accuracy in these studies is high (>95%) and a recent comparative study also suggests that CNNs may out-perform competing supervised classification methods, such as SVMs ( Halicek et al., 2017 ). CNNs can incorporate multiple processing steps within the overall network architecture, including pre-processing calibration steps, within the network, which optimises speed when compared to other regressors such as random forests ( Ayala et al., 2019 ). By avoiding more tailored regression approaches, CNNs may capture subtle signal variations that support interpretation or classification much more effectively.…”

“…More recently, Wirkert et al. (2016) and Ayala et al. (2019) have used MC simulation to quantify the performance of a random forest estimator using varying noise and SNR conditions, and demonstrated improved accuracy data compared to least-squares regression.…”

Spectral imaging enables contrast agent–free real-time ischemia monitoring in laparoscopic surgery

“…This technique removes the arbitrary restriction of recording only three broad spectral bands (RGB) by capturing an n-dimensional feature vector for each camera pixel, where each dimension corresponds to a comparatively narrower spectral band. As different tissue structures have unique optical scattering and absorption properties, knowledge of these optical properties along with spectral measurement data can potentially provide important information on tissue morphology (8)(9)(10) and function (11,12). The term multispectral imaging (MSI) is used when only a few bands (up to dozens) are recorded, while hyperspectral imaging (HSI) refers to hundreds of bands being recorded (7).…”