A mixed reality framework for microsurgery simulation with visual-tactile perception

Xiang, Nan; Liang, Hai-Ning; Yu, Lingyun; Yang, Xiaosong; Zhang, Jian Jun

doi:10.1007/s00371-023-02964-1

Cited by 4 publications

(4 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For instance, Gadwe et al [38] and Liu et al [39] employed 2D fiducial markers to track laparoscopic ultrasound transducer and the cylindrical surgical devices, respectively. Xiang et al [11] proposed the use of 3D trackers to obtain the motion data of real surgical scissors and forceps. The MTIID [40] introduced 3D printed clip-applying forceps combined with the Leap Motion Tracker in a cerebral aneurysm clipping simulation system.…”

Section: Trackingmentioning

confidence: 99%

“…One solution is to attach retro-reflective markers or fiducial markers to surgical instruments or patient tissues, allowing the camera to track the tools based on these markers. Typically, markers come in the form of fiducial markers [11,41] or spherical markers [42]. Fiducial markers are relatively easy to prepare, often involving printing specific shapes on paper and affixing them at predetermined locations for positioning.…”

Section: Vision-based Trackingmentioning

confidence: 99%

“…Synthetic materials such as silicone organ models are readily available but lack visual realism. Biological models, such as cadavers and live animals, are considered better training materials, but they can potentially raise ethical and moral controversies and incur high costs in long-term usage [9][10][11]. In recent years, the rapid development of immersive technologies has made it possible to integrate them into medical training, especially in the field of surgery simulation [12,13].…”

Section: Introductionmentioning

confidence: 99%

“…In VR-based surgical simulation systems, force-feedback devices are typically used as input devices for virtual interaction [4,16]. On the other hand, AR or MR-based systems, in addition to the interaction with force-feedback devices, also employ 3D-tracking methods to capture dynamic data from target objects such as surgical instruments, enabling interaction with the virtual environment [11,17]. Thus, from the foundational principles of interaction and the perspective of algorithms, this study focuses on two core aspects of technology that significantly impact immersive interactive surgical simulation: haptic rendering and tracking.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

Deng,

Xiang,

Pan

2023

Bioengineering

Self Cite

View full text Add to dashboard Cite

Immersive technologies have thrived on a strong foundation of software and hardware, injecting vitality into medical training. This surge has witnessed numerous endeavors incorporating immersive technologies into surgery simulation for surgical skills training, with a growing number of researchers delving into this domain. Relevant experiences and patterns need to be summarized urgently to enable researchers to establish a comprehensive understanding of this field, thus promoting its continuous growth. This study provides a forward-looking perspective by reviewing the latest development of immersive interactive technologies for surgery simulation. The investigation commences from a technological standpoint, delving into the core aspects of virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies, namely, haptic rendering and tracking. Subsequently, we summarize recent work based on the categorization of minimally invasive surgery (MIS) and open surgery simulations. Finally, the study showcases the impressive performance and expansive potential of immersive technologies in surgical simulation while also discussing the current limitations. We find that the design of interaction and the choice of immersive technology in virtual surgery development should be closely related to the corresponding interactive operations in the real surgical speciality. This alignment facilitates targeted technological adaptations in the direction of greater applicability and fidelity of simulation.

show abstract

Section: Trackingmentioning

confidence: 99%

Section: Vision-based Trackingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

Deng,

Xiang,

Pan

2023

Bioengineering

Self Cite

View full text Add to dashboard Cite

show abstract

A Novel Neighbor Aggregation Function for Medical Point Cloud Analysis

Wu,

Qian,

Zheng

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Manufacturing domain instruction comprehension using synthetic data

Johari,

Tong,

Bhardwaj

et al. 2024

Vis Comput

View full text Add to dashboard Cite

Referring Expression Comprehension (REC) system solves a task to localize objects in a given image, based on natural language expression. We propose a novel approach to adapting the pre-trained REC model for the manufacturing domain.Introduction: Despite significant advances in REC research, current REC datasets fail to recognize objects from specific yet important domains such as manufacturing due to the absence of domain-specific samples during training. Thus, we introduce a synthetic data-based domain adaptation approach for REC. Methods:To adapt a REC model to the manufacturing domain, we generated a synthetic REC dataset RefMD that consists of two sub-datasets: 1) dataset for manufacturing object classification, and 2) dataset for REC adaptation to manufacturing. Each dataset serves as one step toward the REC adaptation. Adaptation of the object classification network (visual backbone) is carried out by training ResNet50 on domain-specific labeled data, while the REC adaptation completes with the adaptation of modules in RealGIN altogether. The manufacturing domain-adapted model is further enhanced with the capability to handle ambiguous referring expressions through human-in-the-loop (HITL) interaction. Results:The experiments on 3D printed manufacturing objects demonstrate that the interactive REC model can accurately comprehend human instructions with an 82% accuracy. Conclusion:This paper introduces an approach to facilitate domain adaptation using solely synthetic data for the specific case of the manufacturing domain. However, the proposed adaptation methodology can be applied to any other domain by following the same synthetic data generation process.

show abstract

A mixed reality framework for microsurgery simulation with visual-tactile perception

Cited by 4 publications

References 50 publications

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

A Novel Neighbor Aggregation Function for Medical Point Cloud Analysis

Manufacturing domain instruction comprehension using synthetic data

Contact Info

Product

Resources

About