Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method

Abstract

Background: Worldwide propagation of minimally invasive surgeries (MIS) is hindered by their drawback of indirect observation and manipulation, while monitoring of surgical instruments moving in the operated body required by surgeons is a challenging problem. Tracking of surgical instruments by vision-based methods is quite lucrative, due to its flexible implementation via software-based control with no need to modify instruments or surgical workflow. Methods: A MIS instrument is conventionally split into a shaft and end-effector portions, while a 2D/3D tracking-by-detection framework is proposed, which performs the shaft tracking followed by the end-effector one. The former portion is described by line features via the RANSAC scheme, while the latter is depicted by special image features based on deep learning through a well-trained convolutional neural network. Results: The method verification in 2D and 3D formulation is performed through the experiments on ex-vivo video sequences, while qualitative validation on in-vivo video sequences is obtained. Conclusion: The proposed method provides robust and accurate tracking, which is confirmed by the experimental results: its 3D performance in ex-vivo video sequences exceeds those of the available state-of -the-art methods. Moreover, the experiments on in-vivo sequences demonstrate that the proposed method can tackle the difficult condition of tracking with unknown camera parameters. Further refinements of the method will refer to the occlusion and multi-instrumental MIS applications.

Augmented reality for narrow area navigation in jaw surgery: Modified tracking by detection volume subtraction algorithm

“…However, omissions are errors from erroneous point correspondence when using point cloud, which is a very important factor that needs to be considered for solving the issues of AR navigation when performing dental implants. Zhao et al estimated 3D position and orientation of instruments by using 2D image data with known camera parameters, insertion points, and 2D/3D tracking by a detection framework, performed by tracking the instrument shaft and the end. This method provides robust and accurate tracking of instruments to improve positional.…”

“…However, although the solution provided by Pokhrel et al achieves high accuracy and reduced the error during cutting, positional error reduction and improvement of tracking accuracy were not considered by the authors. To overcome these limitations, we adopted features from the work of Zhao et al…”

“…For jaw surgery, the capability inherent in AR, of layering virtual content onto the field of view of the surgeon, is particularly important to locate the specific areas that need to be cut with a high level of precision. As previously pointed out, there is always a risk of cutting nerve channels or root canals associated with jaw surgery, which makes it desirable to be able to track the position and orientation of surgical instruments used for cutting jaw bones . In terms of technique, a volume subtraction algorithm (VSA) is considered to achieve the most desirable bone‐cutting without markers in low processing time .…”

“…The purpose of this paper is to provide navigational and positional information of the surgical instrument during the jaw surgery. Our proposed solution tracks narrow surgical corridors and makes the anatomical structure beneath the jaw bone visible to ensure accurate positioning of the surgical instrument . The proposed method provides an accurate estimation of the position and orientation of the instrument in 3D.…”

DNN-Based Assistant in Laparoscopic Computer-Aided Palpation

“…Through using DNN, some of these studies detected changes in the mechanical properties of tissue, such as cerebral microbleeds, brain tumors, breast lesions, lung nodules, and pulmonary nodules. Moreover, analysis techniques have been developed where laparoscopic images are used to classify surgical events (Varytimidis et al, 2016;Pestscharing and Schoffmann, 2017), and track (Wang et al, 2017) or classify (Zhao et al, 2017) surgical instruments. These studies targeted postoperative objectives, such as assisting in medical training, analyses, and compiling databases of recorded endoscopic images.…”