A First Evaluation of a Multi-Modal Learning System to Control Surgical Assistant Robots via Action Segmentation

Abstract

The next stage for robotics development is to introduce autonomy and cooperation with human agents in tasks that require high levels of precision and/or that exert considerable physical strain. To guarantee the highest possible safety standards, the best approach is to devise a deterministic automaton that performs identically for each operation. Clearly, such approach inevitably fails to adapt itself to changing environments or different human companions. In a surgical scenario, the highest variability happens for the timing of different actions performed within the same phases. This paper presents a cognitive control architecture that uses a multi-modal neural network trained on a cooperative task performed by human surgeons and produces an action segmentation that provides the required timing for actions while maintaining full phase execution control via a deterministic Supervisory Controller and full execution safety by a velocityconstrained Model-Predictive Controller.

Robotics in laparoscopic surgery - A review

“…A cognitive control architecture is proposed in ref. [189] that integrates the appropriate surgical task determinism using supervisory controller, motion safety using velocity-constrained MPC, and adaptability to human task execution timings provided by action segmentation using multi-modal neural network. The system's overall block diagram is shown in Fig.…”

“…Finally, the MPC receives the current goal and confidence level required to control the SARAS arm, where it is intended to maximize the performance and the safety of the arm incorporating both a collision . Control framework of a multi-modal learning system to control surgical assistant robots via action segmentation [189].…”

“…PosiƟon Control [45,65,71,94,187,189,191,192] Force/Torque Control [190,191] [191] • DisƟnguishes the procedure in progress and determines the next task to be performed (the goal posiƟon) corresponding to each procedure [71,94,189] [ 187,191,192] Figure 18. Classification of control objectives and control inputs of the discussed control systems.…”

“…• MPC + Fuzzy ApproximaƟon to compensate for disturbances [187] • MPC + Supervisory Control and AcƟon SegmentaƟon using mulƟmodal learning [189] Figure 19. Classification of types of controllers used in the discussed control systems.…”

Evaluating the Task Generalization of Temporal Convolutional Networks for Surgical Gesture and Motion Recognition Using Kinematic Data

“…Recent works in gesture recognition have each defined their own sets of gestures for their own datasets [13]- [17] with limited overlap between gestures. On the other hand, works on recognition of fine-grained surgical actions focus on action triplets (verb, instrument, tissue/object) [18]- [20], representing surgical instrument and tissue interactions in endoscopic videos.…”