Augmented Reality Predictive Displays to Help Mitigate the Effects of Delayed Telesurgery

Richter, Florian; Zhang, Yifei; Zhi, Yuheng; Orosco, Ryan K.; Yip, Michael C.

doi:10.1109/icra.2019.8794051

Cited by 34 publications

(14 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The use of virtual fixtures, previously mentioned in the "Training, Skill Assessment, and Gesture Recognition" section as an intraoperative guidance tool, were investigated in [26]- [28], [77], [84], [167], [168], [205], [208], and [209] by providing constraints on the instruments' workspace. The use of force information within augmented reality to provide the surgeons with visual feedback about forces (the socalled pseudohaptics) was also explored [29], [30], [32], [105], [227], [254].…”

Section: Haptics and Pseudohapticsmentioning

confidence: 99%

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

D'Ettorre

Mariani

Stilli

et al. 2021

IEEE Robot. Automat. Mag.

View full text Add to dashboard Cite

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.R obotic-assisted surgery is now well established in clinical practice and has become the gold-standard clinical treatment option for several clinical indications. The field of robotic-assisted surgery is expected to grow substantially in the next decade, with a range of new robotic devices emerging to address unmet clinical needs across different specialties. A vibrant surgical robotics research community is pivotal for conceptualizing such new systems as well as for developing and training the engineers and scientists to translate them into practice. The da Vinci Research Kit (dVRK), an academic and industry collaborative effort to repurpose decommissioned da Vinci surgical systems [Intuitive Surgical Inc. (ISI), California, USA] as a research platform for surgical robotics research, has been a key initiative for addressing a barrier to entry for new research groups in surgical robotics. In this article, we present an extensive review of the publications that have been facilitated by the dVRK over the past decade. We classify research efforts into different categories and outline some of the major challenges and needs for the robotics community to maintain and build upon this initiative.

show abstract

Section: Haptics and Pseudohapticsmentioning

confidence: 99%

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

D'Ettorre

Mariani

Stilli

et al. 2021

IEEE Robot. Automat. Mag.

View full text Add to dashboard Cite

show abstract

“…To fully exploit the prior and enhance the robustness of our system, the surgical tool portion of the image and depth data is not passed to the deformable tissue tracker since the surgical tool is already being tracked. Therefore, a mask of the surgical tool is generated using the same OpenGL rendering pipeline we previously developed [42], and applied to the depth and image data passed to the deformable tissue tracker. To ensure the mask covers all of the tool, it is dilated before being applied.…”

Section: B Depth Map From Stereo Imagesmentioning

confidence: 99%

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics

Richter

et al. 2020

IEEE Robot. Autom. Lett.

Self Cite

View full text Add to dashboard Cite

Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modelled robotic systems in conjunction with multiple sensors. However, application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping of a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical R System in real-time with an endeffector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed autonomous soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research † .

show abstract

“…The organic light-emitting diode (OLED) and liquid crystal display (LCD) do not usually store received pixel data in an intermediate/temporary buffer but do some extra processing, such as re-scaling [28]. In [32], predictive displays found success in overcoming the AR device system latency issue by providing the user with immediate visual feedback.…”

Section: Displaymentioning

confidence: 99%

A Study on Sensor System Latency in VR Motion Sickness

Kundu

Rahman

Paul

2021

JSAN

View full text Add to dashboard Cite

One of the most frequent technical factors affecting Virtual Reality (VR) performance and causing motion sickness is system latency. In this paper, we adopted predictive algorithms (i.e., Dead Reckoning, Kalman Filtering, and Deep Learning algorithms) to reduce the system latency. Cubic, quadratic, and linear functions are used to predict and curve fitting for the Dead Reckoning and Kalman Filtering algorithms. We propose a time series-based LSTM (long short-term memory), Bidirectional LSTM, and Convolutional LSTM to predict the head and body motion and reduce the motion to photon latency in VR devices. The error between the predicted data and the actual data is compared for statistical methods and deep learning techniques. The Kalman Filtering method is suitable for predicting since it is quicker to predict; however, the error is relatively high. However, the error property is good for the Dead Reckoning algorithm, even though the curve fitting is not satisfactory compared to Kalman Filtering. To overcome this poor performance, we adopted deep-learning-based LSTM for prediction. The LSTM showed improved performance when compared to the Dead Reckoning and Kalman Filtering algorithm. The simulation results suggest that the deep learning techniques outperformed the statistical methods in terms of error comparison. Overall, Convolutional LSTM outperformed the other deep learning techniques (much better than LSTM and Bidirectional LSTM) in terms of error.

show abstract

Augmented Reality Predictive Displays to Help Mitigate the Effects of Delayed Telesurgery

Cited by 34 publications

References 27 publications

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

Accelerating Surgical Robotics Research: A Review of 10 Years With the da Vinci Research Kit

SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics

A Study on Sensor System Latency in VR Motion Sickness

Contact Info

Product

Resources

About