Augmented Reality Future Step Visualization for Robust Surgical Telementoring

Abstract

Surgical telementoring connects expert mentors with trainees performing urgent care in austere environments. However, such environments impose unreliable network quality, with significant latency and low bandwidth. We have developed an augmented reality telementoring system that includes future step visualization of the medical procedure. Pregenerated video instructions of the procedure are dynamically overlaid onto the trainee's view of the operating field when the network connection with a mentor is unreliable. METHODS Our future step visualization uses a tablet suspended above the patient's body, through which the trainee views the operating field. Before trainee use, an expert records a "future library" of stepby-step video footage of the operation. Videos are displayed to the trainee as semitransparent graphical overlays. We conducted a study where participants completed a cricothyroidotomy under telementored guidance. Participants used one of two telementoring conditions: conventional telestrator or our system with future step visualization. During the operation, the connection between trainee and mentor was bandwidth throttled. Recorded metrics were idle time ratio, recall error, and task performance. RESULTS Participants in the future step visualization condition had 48% smaller idle time ratio (14.5% vs. 27.9%, P < 0.001), 26% less recall error (119 vs. 161, P = 0.042), and 10% higher task performance scores (rater 1 = 90.83 vs. 81.88, P = 0.008; rater 2 = 88.54 vs. 79.17, P = 0.042) than participants in the telestrator condition. CONCLUSIONS Future step visualization in surgical telementoring is an important fallback mechanism when trainee/mentor network connection is poor, and it is a key step towards semiautonomous and then completely mentor-free medical assistance systems.

Augmented reality practices in health services: Literature review

“…And then Canada is following USA in terms of researches performed on the same [18,20,24,43]. And when the research methods used in the studies were examined, it was observed that literature review [8,9,10,13,14,19,33,34,36,40], compilation and AR practices applied on surgical operations were published at a high degree [17,18,20,24,26,35,44]. (Graph 2) When the fields of application of AR, used in trainings provided in the domain of health, were examined, it is being observed by Graph 3 that it was used the most in the medical education [5,7,8,11,14,22,36,40,43].…”

“…(Graph 2) When the fields of application of AR, used in trainings provided in the domain of health, were examined, it is being observed by Graph 3 that it was used the most in the medical education [5,7,8,11,14,22,36,40,43]. When articles on AR practices used in surgical sciences were examined, it was observed that the highest number of publications was on the domain of General Surgery as clinical domain [13,17,20,44]. And when the key words of the publications were analyzed, it is being observed by Graph 5 that the most frequently used key word was "Augmented Reality" [5,7,8,9,11,13,14,15,16,17,19,20,21,22,23,26,27,29,31,33,34,35,37,38,39,41,42,44,45,46], and the second one was "Virtual Reality" [ 8,9,13,…”

“…When articles on AR practices used in surgical sciences were examined, it was observed that the highest number of publications was on the domain of General Surgery as clinical domain [13,17,20,44]. And when the key words of the publications were analyzed, it is being observed by Graph 5 that the most frequently used key word was "Augmented Reality" [5,7,8,9,11,13,14,15,16,17,19,20,21,22,23,26,27,29,31,33,34,35,37,38,39,41,42,44,45,46], and the second one was "Virtual Reality" [ 8,9,13,14,16,25,27,37,41]. 1 [34] 1 [37] 1 [20] 1 [24] 1 [35] 1…”

Tele-mentoring using augmented reality technology in healthcare: A systematic review

“…Systems accessing the wireless network also need to deal with poor wireless connectivity attributed to the absence of the wireless signal coverage, the weakness of the signal, or the incompatibility between the wireless system and the AR system. In one experiment, Andersen et al (2019) suggested that a future step in visualisation could be a backup mechanism in case of poor network connectivity. For example, before the experiment, the steps of the procedure or performance task could be audio-recorded, and then during the experiment, users could switch between the live instruction provided by the mentor in real time and the instruction recorded previously and stored in the device.…”

Procedural Telementoring in Rural, Underdeveloped, and Austere Settings: Origins, Present Challenges, and Future Perspectives

“…Another problem is that telementoring systems are highly dependent on the telecommunications systems used. As such, the reliability of those systems affects the mentoring process and thus potentially the operative outcome (30), and very few systems can reliably overcome connectivity challenges through fail-safe modes (31). Verbal instruction by the mentor is crucial to telementoring; however, noise in the operating room can interfere with the mentor's audio communication to the extent that the operating surgeon must wear a headset (32).…”