Spatio-temporal deep learning methods for motion estimation using 4D OCT image data

Abstract

Purpose Localizing structures and estimating the motion of a specific target region are common problems for navigation during surgical interventions. Optical coherence tomography (OCT) is an imaging modality with a high spatial and temporal resolution that has been used for intraoperative imaging and also for motion estimation, for example, in the context of ophthalmic surgery or cochleostomy. Recently, motion estimation between a template and a moving OCT image has been studied with deep learning methods to overcome the shortcomings of conventional, feature-based methods. Methods We investigate whether using a temporal stream of OCT image volumes can improve deep learning-based motion estimation performance. For this purpose, we design and evaluate several 3D and 4D deep learning methods and we propose a new deep learning approach. Also, we propose a temporal regularization strategy at the model output. Results Using a tissue dataset without additional markers, our deep learning methods using 4D data outperform previous approaches. The best performing 4D architecture achieves an correlation coefficient (aCC) of 98.58% compared to 85.0% of a previous 3D deep learning method. Also, our temporal regularization strategy at the output further improves 4D model performance to an aCC of 99.06%. In particular, our 4D method works well for larger motion and is robust toward image rotations and motion distortions. Conclusions We propose 4D spatio-temporal deep learning for OCT-based motion estimation. On a tissue dataset, we find that using 4D information for the model input improves performance while maintaining reasonable inference times. Our regularization strategy demonstrates that additional temporal information is also beneficial at the model output.

Random matrix theory tools for the predictive analysis of functional magnetic resonance imaging examinations

“…For example, deep learning methods improve upon guessing by 17% for autism 79 or 3% to 30% for ADHD, 80 16% for severe depression, 81 and 23% for obsessive compulsive disorder 82 . Manual feature engineering with more separable conditions (e.g., schizophrenia) can result in classification accuracies well above 90%, 83 and with larger data, sophisticated custom feature extraction methods can achieve near perfect accuracies at classifying task versus rest 84 .…”

4D spatio-temporal convolutional networks for object position estimation in OCT volumes

“…However, as OCT allows for a temporal stream of OCT image volumes, it seems reasonable that the preceding image volumes at high volume rates may carry information on the object's motion. This leads to the challenging problem of 4D deep learning, which is largely unexplored so far and has only been addressed in a few applications such as functional magnetic resonance imaging [8], computed tomography [9] and OCT-based force estimation [10] as well as OCT-based tissue motion estimation [11].…”

4D Spatio-Temporal Convolutional Networks for Object Position Estimation in OCT Volumes

“…However, as OCT allows for a temporal stream of OCT image volumes, it seems reasonable that the preceding image volumes at high volume rates may carry information on the object's motion. This leads to the challenging problem of 4D deep learning, which is largely unexplored so far and has only been addressed in a few applications such as functional magnetic resonance imaging [1], computed tomography [6] and OCT-based force estimation [8] as well as OCT-based tissue motion estimation [2]. In this work, we systematically extend 3D CNNs to 4D spatio-temporal data processing and evaluate whether a stream of OCT volumes improves object position estimation performance.…”