Automated instrument-tracking for 4D video-rate imaging of ophthalmic surgical maneuvers

Abstract

Intraoperative image-guidance provides enhanced feedback that facilitates surgical decision-making in a wide variety of medical fields and is especially useful when haptic feedback is limited. In these cases, automated instrument-tracking and localization are essential to guide surgical maneuvers and prevent damage to underlying tissue. However, instrument-tracking is challenging and often confounded by variations in the surgical environment, resulting in a trade-off between accuracy and speed. Ophthalmic microsurgery presents additional challenges due to the nonrigid relationship between instrument motion and instrument deformation inside the eye, image field distortion, image artifacts, and bulk motion due to patient movement and physiological tremor. We present an automated instrument-tracking method by leveraging multimodal imaging and deep-learning to dynamically detect surgical instrument positions and re-center imaging fields for 4D video-rate visualization of ophthalmic surgical maneuvers. We are able to achieve resolution-limited tracking accuracy at varying instrument orientations as well as at extreme instrument speeds and image defocus beyond typical use cases. As proof-of-concept, we perform automated instrument-tracking and 4D imaging of a mock surgical task. Here, we apply our methods for specific applications in ophthalmic microsurgery, but the proposed technologies are broadly applicable for intraoperative image-guidance with high speed and accuracy.

Optimization of intraoperative spectrally encoded coherence tomography and reflectometry for ophthalmic surgery

“…Intraoperative optical coherence tomography (iOCT) overcomes limitations in conventional white-light ophthalmic surgical microscopy by providing depth-resolved visualization of ocular microstructures, including epiretinal membranes, macular holes, and corneal layers 1,2 . We previously demonstrated multimodal ophthalmic imaging and instrument tracking using our intraoperative spectrally encoded coherence tomography and reflectometry imaging (iSECTR) system 3,4 . iSECTR uses spatiotemporally co-registered multimodal spectrally encoded reflectometry (SER) and OCT for automated en face instrument-tracking and volumetric visualization of surgical dynamics 5…”

Surgical Tool Datasets for Machine Learning Research: A Survey

“…Tang et al (2022) leveraged multimodal imaging and deeplearning to dynamically detect surgical instrument positions in ophthalmic surgical maneuvers. In their system, they combined spectrally encoded reflectometry (SER) and crosssectional OCT imaging for automated instrument-tracking, and tested it on 4730 manually-labelled SER images of a 25-gauge internal limiting membrane (25 G ILM) forceps.Al defined tool tracking work in terms of monitoring tool location over time Gruijthuijsen et al (2021).…”