Evaluating a PSMNet-based pipeline for intraoperative stereo reconstruction with the da Vinci Xi

Abstract

Stereo reconstruction is an important tool for generating 3D surface observations of deformable tissues that can be used to non-rigidly update intraoperative image guidance. As compared to traditional image processing-based stereo matching techniques, emerging machine learning approaches aim to deliver shorter processing times, more accurate surface reconstructions, and greater robustness to the suboptimal qualities of intraoperative tissue imaging (e.g., occlusion, reflection, and minimally textured surfaces). This work evaluates the popular PSMNet convolutional neural network as tool for generating disparity maps from the video feed of the da Vinci Xi Surgical System. Reconstruction accuracy and speed were assessed for a series of 44 stereoendoscopic frame pairs showing key structures in a silicone renal phantom. Surface representation accuracy was found to be on the order of 1mm for reconstructions of the kidney and inferior vena cava, and disparity maps were produced in under 2s when inference was performed on a standard modern GPU. These preliminary results suggest that PSMNet and similar trained models may be useful tools for integrating intraoperative stereo reconstruction into advanced navigation platforms and warrant further development of the overall data pipeline and testing with biological tissues in representative surgical conditions.

Initial implementation of robot-integrated specimen imaging in transoral robotic surgery

“…In post processing, the stereo video streams were manually reviewed and selected frames were flagged for automatic extraction using custom FFmpeg-based utilities. Stereo pairs were rectified in MATLAB and dense disparity maps were estimated using PSMNet [10,11] on a 3.50 GHz Intel ® Core TM i9-11900K PC running Windows 10. CCTag locations were identified in each rectified frame using software available 1 from the original authors, allowing direct computation of disparity values for each marker.…”

A robust system for capture and archival of high-definition stereoendoscopic video

“…The increasing utilization of stereoscopic imaging in surgery also provides an opportunity for developing stereoupdated image guidance, where contours of the surgical field are captured in real time via stereo reconstruction and used to generate an estimated warping of the relevant preoperative tomography. Applications span surgical subspecialties and anatomical sites including the brain [1], heart [2], urinary tract [3,4], and spine [5], among others.…”