Hand Motion-Aware Surgical Tool Localization and Classification from an Egocentric Camera

Abstract

Detecting surgical tools is an essential task for the analysis and evaluation of surgical videos. However, in open surgery such as plastic surgery, it is difficult to detect them because there are surgical tools with similar shapes, such as scissors and needle holders. Unlike endoscopic surgery, the tips of the tools are often hidden in the operating field and are not captured clearly due to low camera resolution, whereas the movements of the tools and hands can be captured. As a result that the different uses of each tool require different hand movements, it is possible to use hand movement data to classify the two types of tools. We combined three modules for localization, selection, and classification, for the detection of the two tools. In the localization module, we employed the Faster R-CNN to detect surgical tools and target hands, and in the classification module, we extracted hand movement information by combining ResNet-18 and LSTM to classify two tools. We created a dataset in which seven different types of open surgery were recorded, and we provided the annotation of surgical tool detection. Our experiments show that our approach successfully detected the two different tools and outperformed the two baseline methods.

Computer vision in surgery: from potential to clinical value

“…While standardized measurements with predictable variability can be utilized in tabular data, such as laboratory values for hemoglobin or creatinine, defining clinical phenomena in surgical videos (i.e., annotation) can be quite difficult. Open surgery presents unique challenges that occur with occlusion of video data from the surgeon’s own movements, necessitating multiple camera angles, additional sensors, or algorithmic approaches to overcome occlusion and consider the added complexity of hand-tool interactions 54 – 56 .…”

Open surgery tool classification and hand utilization using a multi-camera system

“…For example, a Close-up view may assist in identifying the different anatomical landmarks as well as help distinguish between similar tools. While a far away top view may offer a view of the tool tray as well as the assistant's hand [9].…”

“…In the computer vision field, the task of object detection is well studied [10,11]. In the medical field, most studies focus on MIS [4,7,8,12], with only few focusing on tool detection in open surgery [9,13,14]. Several works have used multi-angle photography during MIS, with the internal camera capturing the tools being used, and the other cameras identifying the area surrounding the surgery.…”

“…Furthermore, if a low fps is used a longer time window may be used while sustaining a reasonable input size. Several tool classification studies evaluated sequence length or specific fps to optimize results [9,[20][21][22]. In this study we chose a dual LSTM [23] system.…”

Deep Learning in Medical Image Analysis

“…Shimizu and Hachiuma (2021) [ 6 ] from Japan combine three modules for localization, selection, and classification for the detection of the two surgical tools. In the localization module, the authors employ the Faster R-CNN to detect surgical tools and target hands, and in the classification module, the authors extract hand movement information by combining ResNet-18 and long short-term memory (LSTM) to classify two tools.…”