Language-based translation and prediction of surgical navigation steps for endoscopic wayfinding assistance in minimally invasive surgery

Abstract

Purpose In the context of aviation and automotive navigation technology, assistance functions are associated with predictive planning and wayfinding tasks. In endoscopic minimally invasive surgery, however, assistance so far relies primarily on image-based localization and classification. We show that navigation workflows can be described and used for the prediction of navigation steps. Methods A natural description vocabulary for observable anatomical landmarks in endoscopic images was defined to create 3850 navigation workflow sentences from 22 annotated functional endoscopic sinus surgery (FESS) recordings. Resulting FESS navigation workflows showed an imbalanced data distribution with over-represented landmarks in the ethmoidal sinus. A transformer model was trained to predict navigation sentences in sequence-to-sequence tasks. The training was performed with the Adam optimizer and label smoothing in a leave-one-out cross-validation study. The sentences were generated using an adapted beam search algorithm with exponential decay beam rescoring. The transformer model was compared to a standard encoder-decoder-model, as well as HMM and LSTM baseline models. Results The transformer model reached the highest prediction accuracy for navigation steps at 0.53, followed by 0.35 of the LSTM and 0.32 for the standard encoder-decoder-network. With an accuracy of sentence generation of 0.83, the prediction of navigation steps at sentence-level benefits from the additional semantic information. While standard class representation predictions suffer from an imbalanced data distribution, the attention mechanism also considered underrepresented classes reasonably well. Conclusion We implemented a natural language-based prediction method for sentence-level navigation steps in endoscopic surgery. The sentence-level prediction method showed a potential that word relations to navigation tasks can be learned and used for predicting future steps. Further studies are needed to investigate the functionality of path prediction. The prediction approach is a first step in the field of visuo-linguistic navigation assistance for endoscopic minimally invasive surgery.

A Scoping Review of Artificial Intelligence Research in Rhinology

“…Of the 79 articles identified, 3 were published in 2017, 17–19 5 were published in 2018, 20–24 9 were published in 2019, 25–33 15 were published in 2020, 34–48 31 were published in 2021, 49–79 and 16 were published in 2022 80…”

Keyword-augmented and semi-automatic generation of FESS reports: a proof-of-concept study

“…Therefore, we are simultaneously developing an image-based tool that can recognize and analyze endonasal endoscopic videos with the help of a pattern recognition software. That software was already reported in ORL procedures [ 2 , 7 ]. Google® already uses similar technologies to provide video content information [ 2 ] automatically.…”

“…These enable the generation of a complete and semantically logical surgical report. Regarding that topic, improvements in the field of image- and text-based production of reports based on NLP and deep learning processes were shown [ 4 , 7 ]. Those previous works set the foundation for generating radiology reports (4) and prediction of navigation steps during endoscopic operations by automatically analyzing image structures and datasets (7).…”

BIOPASS-Hybrid-Navigation für die endoskopische Nasennebenhöhlenchirurgie – ein Assistenzsystem

“…Die Lage-, Bewegungs-und Abstandsmessung zeigte dem Teilnehmer die Tiefe im Situs in mm an. Das Software-Modul gab bei Erreichen einer Landmarke sowohl per Sprachausgabe wie auch per Text im Endoskopiebild den Namen der Landmarke an [24]. Zudem schlug es in Textform die nächste mögliche Landmarke vor (▶ Abb.…”