Use of Commercial Off-The-Shelf Devices for the Detection of Manual Gestures in Surgery: Systematic Literature Review

Abstract

Background The increasingly pervasive presence of technology in the operating room raises the need to study the interaction between the surgeon and computer system. A new generation of tools known as commercial off-the-shelf (COTS) devices enabling touchless gesture–based human-computer interaction is currently being explored as a solution in surgical environments. Objective The aim of this systematic literature review was to provide an account of the state of the art of COTS devices in the detection of manual gestures in surgery and to identify their use as a simulation tool for motor skills teaching in minimally invasive surgery (MIS). Methods For this systematic literature review, a search was conducted in PubMed, Excerpta Medica dataBASE, ScienceDirect, Espacenet, OpenGrey, and the Institute of Electrical and Electronics Engineers databases. Articles published between January 2000 and December 2017 on the use of COTS devices for gesture detection in surgical environments and in simulation for surgical skills learning in MIS were evaluated and selected. Results A total of 3180 studies were identified, 86 of which met the search selection criteria. Microsoft Kinect (Microsoft Corp) and the Leap Motion Controller (Leap Motion Inc) were the most widely used COTS devices. The most common intervention was image manipulation in surgical and interventional radiology environments, followed by interaction with virtual reality environments for educational or interventional purposes. The possibility of using this technology to develop portable low-cost simulators for skills learning in MIS was also examined. As most of the articles identified in this systematic review were proof-of-concept or prototype user testing and feasibility testing studies, we concluded that the field was still in the exploratory phase in areas requiring touchless manipulation within environments and settings that must adhere to asepsis and antisepsis protocols, such as angiography suites and operating rooms. Conclusions COTS devices applied to hand and instrument gesture–based interfaces in the field of simulation for skills learning and training in MIS could open up a promising field to achieve ubiquitous training and presurgical warm up.

Evaluating Usability of a Touchless Image Viewer in the Operating Room

“…This pilot study describes the usability evaluation of a prototype touchless medical image viewer by surgeons in the operating room. Since the general concept of the "Touchless Operating Room" is not novel and several studies examined the use of hand gestures as a mode of touchless interaction before, 6,8 we sought to focus our study on evaluating the system in a modeled close-to-real scenario by end users, because the need for usability testing has been pointed out previously. 6,17,21,22 Compared with previous studies we chose a different mode of human-computer interaction with a different set of gestures.…”

“…Previous systems used timeof-flight (TOF) cameras and segmentation to emulate a pointing device which allowed for different user interfaces. 5,8,9 Our system was specifically designed to use standard PC hardware without the need for TOF cameras and is able to run on a laptop with a front-facing webcam, thus reducing investment cost. However, using a bigger screen and a camera mounted at eye level improved accuracy at greater distances from the screen during initial testing.…”

“…Both approaches have been demonstrated to be feasible and effective. 8,9 In 2017, Mewes et al conducted a systematic review of available evidence for the use of touchless human-computer interaction in sterile, medical environments. 6 In their study, the majority of systems (34 studies or 62% of all examined studies) described systems to control medical image viewers.…”

Use of a Low-Cost Portable 3D Virtual Reality Gesture-Mediated Simulator for Training and Learning Basic Psychomotor Skills in Minimally Invasive Surgery: Development and Content Validity Study

“…The instrument under investigation is a contactless, gesture-mediated simulator that uses the LMC (construct context). To determine the current use of gesture-mediated interfaces in surgery, especially in the field of surgical simulation, a systematic literature review was conducted [ 14 ]. Finally, as content-related validity evidence was collected, the goal was to identify whether there were any areas of construct underrepresentation or construct irrelevance.…”

“…Finally, from 2002 onward, the evidence for concurrent validity was also demonstrated [ 12 , 13 ]. Recent years have seen the development of low-cost, gesture-based touchless devices that can interact with 3D virtual environments, such as the Microsoft Kinect (MS Kinect, Microsoft Corp), Leap Motion Controller (LMC; Leap Motion Inc), and the Myo armband (Thalmic Labs, Kitchener) [ 14 ].…”