DESK: A Robotic Activity Dataset for Dexterous Surgical Skills Transfer to Medical Robots

Abstract

Datasets are an essential component for training effective machine learning models. In particular, surgical robotic datasets have been key to many advances in semiautonomous surgeries, skill assessment, and training. Simulated surgical environments can enhance the data collection process by making it faster, simpler and cheaper than real systems. In addition, combining data from multiple robotic domains can provide rich and diverse training data for transfer learning algorithms. In this paper, we present the DESK (Dexterous Surgical Skill) dataset. It comprises a set of surgical robotic skills collected during a surgical training task using three robotic platforms: the Taurus II robot, Taurus II simulated robot, and the YuMi robot. This dataset was used to test the idea of transferring knowledge across different domains (e.g. from Taurus to YuMi robot) for a surgical gesture classification task with seven gestures. We explored three different scenarios: 1) No transfer, 2) Transfer from simulated Taurus to real Taurus and 3) Transfer from Simulated Taurus to the YuMi robot. We conducted extensive experiments with three supervised learning models and provided baselines in each of these scenarios. Results show that using simulation data during training enhances the performance on the real robot where limited real data is available. In particular, we obtained an accuracy of 55% on the real Taurus data using a model that is trained only on the simulator data. Furthermore, we achieved an accuracy improvement of 34% when 3% of the real data is added into the training process.

Gesture Recognition in Robotic Surgery: A Review

“…surgical tool kinematics or semantic segmentation of the 3D scene) to aid recognition. Further efforts could then be directed towards new strategies for knowledge transfer between surgical environments (simulation vs real surgery), as well as between surgical tasks [76,30] or surgical robots [91], in order to exploit cross-domain information to improve generalization performance.…”

Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?

“…Although various datasets are available in the public domain (as discussed earlier), a very small number of these datasets are designed for autonomous or semi-autonomous surgery. Such datasets are very important for advancing research in AIS and semi-autonomous surgeries [ 90 ]. An example of an attempt to prepare a dedicated dataset for AIS is the work presented in [ 91 ], which presented an experimental framework to evaluate the performance of CV methods in analyzing operative videos to identify steps of a laparoscopic sleeve gastrectomy (LSG).…”

Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment

“…The dataset we utilized is Dexterous Surgical Skills Transfer (DESK) dataset [21], which contains peg transfer motions recorded from a simulated Taurus II robot, and a real Taurus II robot. The motion generally consists of picking an object from a pegboard with one robotic arm, transferring it to the other arm, and positioning the object over a target peg on the opposite side of the board.…”

“…The dataset we utilize is the Desk dataset [21], the peg transfer task in Desk dataset has the following variations that could lead to kinematic or visual domain shift between simulator and real robot: 1) Initial and final positions of objects on the pegboard, 2) direction of the transfer(objects on the left side are transferred to the right and vice versa) and 3) position/orientation of the pegboard(left/right) maneuvers. From the DESK dataset, we used Taurus simulator video sequences of S1-S5, Taurus real robot S1-S8.…”