Towards Retrieving Force Feedback in Robotic-Assisted Surgery: A Supervised Neuro-Recurrent-Vision Approach

Abstract

Robotic-assisted minimally invasive surgeries have gained a lot of popularity over conventional procedures as they offer many benefits to both surgeons and patients. Nonetheless, they still suffer from some limitations that affect their outcome. One of them is the lack of force feedback which restricts the surgeon's sense of touch and might reduce precision during a procedure. To overcome this limitation, we propose a novel force estimation approach that combines a vision based solution with supervised learning to estimate the applied force and provide the surgeon with a suitable representation of it. The proposed solution starts with extracting the geometry of motion of the heart's surface by minimizing an energy functional to recover its 3D deformable structure. A deep network, based on a LSTM-RNN architecture, is then used to learn the relationship between the extracted visual-geometric information and the applied force, and to find accurate mapping between the two. Our proposed force estimation solution avoids the drawbacks usually associated with force sensing devices, such as biocompatibility and integration issues. We evaluate our approach on phantom and realistic tissues in which we report an average root-mean square error of 0.02 N.

Needle Tip Force Estimation Using an OCT Fiber and a Fused convGRU-CNN Architecture

“…For our model input, we consider a series of A-scans prior to the current observation, as the current force estimate likely depends on prior deformation [1]. Furthermore, we do not extract the epoxy surface as an explicit deformation feature but instead let our model learn relevant features.…”

Sequential Image-Based Attention Network for Inferring Force Estimation Without Haptic Sensor

“…Studies have been conducted to measure interaction forces without force sensors. In [2], a stereo camera was used to reconstruct a 3D artificial heart surface and a supervised learning method was applied to predict the applied force. In [38], a video-based method to estimate the interaction force between a human body and an object was proposed using 3D modeling information.…”

“…The main contributions of this paper are as follows: (1) We propose a computational method for predicting the haptic interaction force only from visual information without a haptic sensor. (2) The sequential image-based attention modules are proposed for efficiently processing the increased convolutional features due to the sequential images and for obtaining more accurate haptic information at the same time.…”

Force estimation from OCT volumes using 3D CNNs

“…More recent approaches have combined template matching and machine learning models [17,18]. Recently, recurrent neural networks (RNN) have been proposed to learn forces based on deformation tracked over time [19,20]. The tissue surface is reconstructed from stereoscopic camera images and features representing surface deformation are defined.…”