Spatio-temporal deep learning models for tip force estimation during needle insertion

Abstract

PurposePrecise placement of needles is a challenge in a number of clinical applications such as brachytherapy or biopsy. Forces acting at the needle cause tissue deformation and needle deflection which in turn may lead to misplacement or injury. Hence, a number of approaches to estimate the forces at the needle have been proposed. Yet, integrating sensors into the needle tip is challenging and a careful calibration is required to obtain good force estimates.MethodsWe describe a fiber-optic needle tip force sensor design using a single OCT fiber for measurement. The fiber images the deformation of an epoxy layer placed below the needle tip which results in a stream of 1D depth profiles. We study different deep learning approaches to facilitate calibration between this spatio-temporal image data and the related forces. In particular, we propose a novel convGRU-CNN architecture for simultaneous spatial and temporal data processing.ResultsThe needle can be adapted to different operating ranges by changing the stiffness of the epoxy layer. Likewise, calibration can be adapted by training the deep learning models. Our novel convGRU-CNN architecture results in the lowest mean absolute error of and a cross-correlation coefficient of 0.9997 and clearly outperforms the other methods. Ex vivo experiments in human prostate tissue demonstrate the needle’s application.ConclusionsOur OCT-based fiber-optic sensor presents a viable alternative for needle tip force estimation. The results indicate that the rich spatio-temporal information included in the stream of images showing the deformation throughout the epoxy layer can be effectively used by deep learning models. Particularly, we demonstrate that the convGRU-CNN architecture performs favorably, making it a promising approach for other spatio-temporal learning problems.Electronic supplementary materialThe online version of this article (10.1007/s11548-019-02006-z) contains supplementary material, which is available to authorized users.

Developing a novel force forecasting technique for early prediction of critical events in robotics

“…sensor and the optical coherence tomography (OCT) using deep convolutional neural networks [32]. Their method demonstrated real-time capability due to low computation times.…”

Needle tip force estimation by deep learning from raw spectral OCT data

“…We acquire calibration data for three custom build needles identical in construction. The data acquisition is performed similar to [3] where a needle is driven against a flat surface with a stepper motor. The force in the axial direction of the needle is measured with a force sensor and recorded together with the associated raw OCT data.…”

“…Here, we consider a setting where optical coherence tomography is B M. Gromniak martin.gromniak@tuhh.de 1 Institute of Medical Technology, Hamburg University of Technology, Hamburg, Germany available, e.g., to study tissue deformation [10] or to realize elastography [7]. While OCT has been proposed for tip forces estimation before [2,3], these approaches rely on the reconstructed gray value data. However, using the reconstructed data has two limitations.…”

Rupture Detection During Needle Insertion Using Complex OCT Data and CNNs

“…Force sensing at the needle tip [23] has been facilitated with a combination of OCT and deep learning methods. Nevertheless, this approach only allows to analyze the motion of an epoxy resin layer attached to the needle tip, but tissue imaging in front of the needle has not been considered.…”

“…Nevertheless, this approach only allows to analyze the motion of an epoxy resin layer attached to the needle tip, but tissue imaging in front of the needle has not been considered. Applications using the different OCT needle designs [19]- [23] require a fast processing and classification of the OCT image data for online control which is challenging due to the high amount of data and the tissue dependent signal characteristics.…”