Real-Time Context-Aware Detection of Unsafe Events in Robot-Assisted Surgery

Abstract

Cyber-physical systems for robotic surgery have enabled minimally invasive procedures with increased precision and shorter hospitalization. However, with increasing complexity and connectivity of software and major involvement of human operators in the supervision of surgical robots, there remain significant challenges in ensuring patient safety. This paper presents a safety monitoring system that, given the knowledge of the surgical task being performed by the surgeon, can detect safety-critical events in real-time. Our approach integrates a surgical gesture classifier that infers the operational context from the time-series kinematics data of the robot with a library of erroneous gesture classifiers that given a surgical gesture can detect unsafe events. Our experiments using data from two surgical platforms show that the proposed system can detect unsafe events caused by accidental or malicious faults within an average reaction time window of 1,693 milliseconds and F1 score of 0.88 and human errors within an average reaction time window of 57 milliseconds and F1 score of 0.76. I. INTRODUCTIONRobot-assisted surgery (RAS) is now a standard procedure across various surgical specialties, including gynecology, urology and general surgeries. During the last two decades, over 2 million procedures were performed using the Intuitive Surgical's daVinci robot in the U.S. [1]. Surgical robots are complex human-in-the-loop Cyber-Physical Systems (CPS) that enable 3D visualization of surgical field and more precise manipulation of surgical instruments such as scissors, graspers, and electro-cautery inside patient's body. The current generation of surgical robots are not fully autonomous yet. They are in level 0 of autonomy [2], following the commands provided by the surgeons from a master-side teleoperation console in real-time (Figure 1a), translating them into precise movements of robotic arms and instruments, while scaling surgeon's motions and filtering out handtremors. By increasing flexibility and precision, surgical robots have enabled new types of surgical procedures and have reduced complication rates and procedure times.Recent studies have shown that safety in robotic surgery may be compromised by vulnerabilities of the surgical robots to accidental or maliciously-crafted faults in the cyber or physical layers of the control system or human errors [3], [4]. Examples of system faults include disruptions of the communication between the surgeon console and the robot, causing packet drops or delays in tele-operation [5], accidental or malicious faults targeting the robot control software [6],

Gesture Recognition in Robotic Surgery: A Review

“…As for the lookahead time, the majority of reported methods relies on a variable number of future frames for robust recognition. While real-time performance or ability to predict future samples are not always needed and pursued, further efforts in this direction could broaden the applicability of such algorithms within the surgical theatre, supporting systems for surgical process monitoring and anomaly detection and prevention [21,61].…”

“…Finally, an emerging and closely related research sub-field is represented by the detection and forecast of surgical errors in fine-grained gestures. It has been recently shown that common gesture-specific errors can be identified in real-time if provided with corresponding real-time gesture labels [21]. Public availability of error annotations [21] will hopefully stimulate future research in this relatively unexplored field, which is fundamental for several CAI implementations including workflow monitoring and surgical automation.…”

“…Beyond off-line generation of training datasets for surgical automation and training, automatic recognition can be exploited for online applications where context-awareness is essential, such as systems for surgical process monitoring, error detection and intra-operative guidance and assistance of different kinds (e.g. robotic, visual, haptic) [21,22]. This will have a potentially high impact in robotic surgery, where demands for increased safety and effectiveness are particularly strong [23].…”

Analysis of Executional and Procedural Errors in Dry-lab Robotic Surgery Experiments

“…With advances in sensing and computing technology, artificial intelligence, and data science, the next generation of Robot-Assisted Surgery (RAS) systems is envisioned to benefit from new capabilities for context-specific monitoring [31] and virtual coaching during simulation training as well as decision support and cognitive assistance during actual surgery to improve safety, efficiency, and quality of care [28]. State-of-the-art RAS systems and simulators are designed with data logging mechanisms to collect system logs, kinematics, and video data from surgical procedures.…”

“…Our goal is to augment RAS systems and simulators with mechanisms to monitor the progress of surgical tasks, and provide early and context-specific feedback to surgeons on potentially suboptimal or unsafe motions that might lead to low performance scores in training or safety-critical events during surgery [31]. In this study, we take a step towards this goal by analyzing recorded dry-lab demonstrations of two common tasks (Suturing and Needle Passing) performed on the da Vinci Research Kit (dVRK) [14].…”