Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery

Wen, Rong; Chng, Chin-Boon; Chui, Chee-Kong

doi:10.3390/robotics6020013

Cited by 25 publications

(19 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When evaluating the Augmented Reality Tracking Error on six patient-specific phantoms the overall error was 2.80 mm. These results are similar to those reported in [ 34 ], using Microsoft HoloLens, and in [ 44 ], using a tablet-based system. However, a relevant fact is that in [ 34 ] the AR marker was two-dimensional, limiting the mobility of the AR device.…”

Section: Discussionsupporting

confidence: 91%

Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation

Moreta-Martínez

Pose-Díez-de-la-Lastra

Calvo-Haro

et al. 2021

Sensors

View full text Add to dashboard Cite

During the last decade, orthopedic oncology has experienced the benefits of computerized medical imaging to reduce human dependency, improving accuracy and clinical outcomes. However, traditional surgical navigation systems do not always adapt properly to this kind of interventions. Augmented reality (AR) and three-dimensional (3D) printing are technologies lately introduced in the surgical environment with promising results. Here we present an innovative solution combining 3D printing and AR in orthopedic oncological surgery. A new surgical workflow is proposed, including 3D printed models and a novel AR-based smartphone application (app). This app can display the patient’s anatomy and the tumor’s location. A 3D-printed reference marker, designed to fit in a unique position of the affected bone tissue, enables automatic registration. The system has been evaluated in terms of visualization accuracy and usability during the whole surgical workflow. Experiments on six realistic phantoms provided a visualization error below 3 mm. The AR system was tested in two clinical cases during surgical planning, patient communication, and surgical intervention. These results and the positive feedback obtained from surgeons and patients suggest that the combination of AR and 3D printing can improve efficacy, accuracy, and patients’ experience.

show abstract

Section: Discussionsupporting

confidence: 91%

Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation

Moreta-Martínez

Pose-Díez-de-la-Lastra

Calvo-Haro

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Table ) showing that some clinical applications had much wider representation within our sample than others. (Maruyama et al, 2018, Zhang, Chen and Liao, 2017, Wen, Chng and Chui, 2017, Sun et al, 2017, Scolozzi and Bijlenga, 2017, Drouin S. et al, 2017, Hou et al, 2016, Zhang X., Chen and Liao, 2015, Yoshino et al, 2015, Kramers et al, 2014, Han et al, 2013, Mahvash and Tabrizi, 2013, Müller M. et al, 2013, Kersten-Oertel et al, 2012, Volonte et al, 2011, Sugimoto et al, 2010, Giraldez et al, 2007 Indicate correct entry points and trajectories of surgical instruments 31.58 24 (Andress et al, 2018, Eftekhar, 2016, Fichtinger et al, 2005, Gavaghan et al, 2012, Khan et al, 2006, Lee J.-D. et al, 2010, Liao et al, 2010, Martins et al, 2016, Rodriguez et al, 2012, Wesarg et al, 2004 Indicate correct soft tissue resection margins and osteotomy lines 21.05 16 (Badiali et al, 2014, Besharati Tabrizi and Mahvash, 2015, Lin et al, 2016, Lin et al, 2015, Mondal et al, 2015, Pessaux et al, 2015, Qu et al, 2015, Shao et al, 2014, Sun et al, 2016…”

Section: Resultsmentioning

confidence: 99%

“…matching two sets of 3D points corresponding to the position of markers on the patient's body and their corresponding points on the patient's scans (Ma et al, 2019). Some studies used computer tracking libraries and/or Software Development Kits (SDKs) (Cutolo et al, 2016, Zeng et al, 2017, such as OpenCV (Shao et al, 2014), ARToolkit (http://www.hitl.washington.edu/artoolkit/) (Lin et al, 2016, Qu et al, 2015 or Vuforia SDK (https://www.vuforia.com/) (Kramers et al, 2014, Wen, Chng andChui, 2017). Both ARToolkit and Vuforia SDK provide algorithms to track 2D and 3D feature points on images and define a shared coordinate system between the digital data and the real world (e.g.…”

Section: Registration Methodsmentioning

confidence: 99%

“…% N Custom algorithms 56.90 33 (Badiali et al, 2014, Giraldez et al, 2007, He, Liu and Wang, 2016, Hu, Wang and Song, 2013, Liao et al, 2010, Lin et al, 2015, Ma et al, 2019, Maruyama et al, 2018, 10 Shamir et al, 2011, Yoshino et al, 2015, Zhang, Chen and Liao, 2017, Zhang, Chen and Liao, 2015 Fully integrated platforms 15.52 9 , Khan et al, 2006, Sun et al, 2017, Wesarg et al, 2004 Tracking libraries/SDKs 20.69 12 (Gavaghan et al, 2012, Kersten-Oertel et al, 2012, Kramers et al, 2014, Lin et al, 2016, Qu et al, 2015, Shao et al, 2014, Wen, Chng and Chui, 2017, Zeng et al, 2017…”

Section: Studies Articlesmentioning

confidence: 99%

See 1 more Smart Citation

Image Overlay Surgery Based on Augmented Reality: A Systematic Review

Pérez-Pachón

Poyade

Lowe

et al. 2020

Advances in Experimental Medicine and Biology

View full text Add to dashboard Cite

Augmented Reality (AR) applied to surgical guidance is gaining relevance in clinical practice. ARbased image overlay surgery (i.e. the accurate overlay of patient-specific virtual images onto the body surface) helps surgeons to transfer image data produced during the planning of the surgery (e.g. the correct resection margins of tissue flaps) to the operating room, thus increasing accuracy and reducing surgery times. We systematically reviewed 76 studies published between 2004 and August 2018 to explore which existing tracking and registration methods and technologies allow healthcare professionals and researchers to develop and implement these systems in-house. Most studies used non-invasive markers to automatically track a patient's position, as well as customised algorithms, tracking libraries or software development kits (SDKs) to compute the registration between patient-specific 3D models and the patient's body surface. Few studies combined the use of holographic headsets, SDKs and user-friendly game engines, and described portable and wearable systems that combine tracking, registration, hands-free navigation and direct visibility of the surgical site. Most accuracy tests included a low number of subjects and/or measurements and did not normally explore how these systems affect surgery times and success rates. We highlight the need for more procedure-specific experiments with a sufficient number of subjects and measurements and including data about surgical outcomes and patients' recovery. Validation of systems combining the use of holographic headsets, SDKs and game engines is especially interesting as this approach allows to easily develop mobile AR applications, thus facilitating the implementation of AR-based image overlay surgery in clinical practice.

show abstract

“…This complexity of the SAR image is a big problem for a quality segmentation. Smoothed and uncomplicated SAR images help more quality and high accuracy segmentation [4]. Recent studies show that the researchers focus on the SAR image despeckling.…”

Section: Introductionmentioning

confidence: 99%

A New Classification Approach with Deep Mask R-CNN for Synthetic Aperture Radar Image Segmentation

Yayla

Şen

2020

ELEKTRON ELEKTROTECH

View full text Add to dashboard Cite

In this paper, a hybrid classification approach which is combined with a more deep mask region-convolutional neural network and sparsity driven despeckling algorithm is proposed for synthetic aperture radar (SAR) image segmentation instead of the classical segmentation methods. In satellite technology, synthetic aperture radar images are strongly used for a lot of areas, such as evaluating air conditions, determining agricultural fields, climatic changes, and as a target in the military. Synthetic aperture radar images must be segmented to each meaningful point in the image for a quality segmentation process. In contrast, synthetic aperture radar images have a lot of noisy speckles and these speckles should be also reduced for a quality segmentation. Current studies show that deep learning techniques are widely used for segmentation methods. High accuracy and fast results can be obtained with deep learning techniques for image segmentation. Mask region-convolutional neural network can not only separate each meaningful field in the image, but it can also generate a high accuracy prediction for each meaningful field of synthetic aperture radar images. The study shows that smoothed SAR images can be classified as multiple regions with deep neural networks.

show abstract

Augmented Reality Guidance with Multimodality Imaging Data and Depth-Perceived Interaction for Robot-Assisted Surgery

Cited by 25 publications

References 27 publications

Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation

Combining Augmented Reality and 3D Printing to Improve Surgical Workflows in Orthopedic Oncology: Smartphone Application and Clinical Evaluation

Image Overlay Surgery Based on Augmented Reality: A Systematic Review

A New Classification Approach with Deep Mask R-CNN for Synthetic Aperture Radar Image Segmentation

Contact Info

Product

Resources

About