3D reconstruction of cystoscopy videos for comprehensive bladder records

Abstract

White light endoscopy is widely used for diagnostic imaging of the interior of organs and body cavities, but the inability to correlate individual 2D images with 3D organ morphology limits its utility for quantitative or longitudinal studies of disease physiology or cancer surveillance. As a result, most endoscopy videos, which carry enormous data potential, are used only for real-time guidance and are discarded after collection. We present a computational method to reconstruct and visualize a 3D model of organs from an endoscopic video that captures the shape and surface appearance of the organ. A key aspect of our strategy is the use of advanced computer vision techniques and unmodified, clinical-grade endoscopy hardware with few constraints on the image acquisition protocol, which presents a low barrier to clinical translation. We validate the accuracy and robustness of our reconstruction and co-registration method using cystoscopy videos from tissue-mimicking bladder phantoms and show clinical utility during cystoscopy in the operating room for bladder cancer evaluation. As our method can powerfully augment the visual medical record of the appearance of internal organs, it is broadly applicable to endoscopy and represents a significant advance in cancer surveillance opportunities for big-data cancer research.

Current Standards in the Endoscopic Management of Bladder Cancer: A Survey Evaluation among Urologists in German-Speaking Countries

“…Similarly, written surgical reports are the main method (77%) used to communicate endoscopic findings between urologists for further surveillance referral -significantly more in the inpatient setting. Numerous studies, however, have described written reports and the bladder diagram as suboptimal and suggest they are not only a major limitation for patient follow-up and treatment but may also play a role in the high recurrence rates [5,6,[21][22][23]. The current documentation approach lacks anatomical accuracy due to the inter-and intra-urologist variability related to the diagram and interpretation of macroscopic findings [5].…”

“…The current documentation approach lacks anatomical accuracy due to the inter-and intra-urologist variability related to the diagram and interpretation of macroscopic findings [5]. To overcome these challenges, a registration/navigation system [5], photo documentation [23], and 3D reconstruction of cystoscopic videos have been proposed Urol Int 2020;104:410-416 DOI: 10.1159/000506653 [6,21]. Such strategies would facilitate clinical history reconstruction, as an increase in time from the initial presentation makes it progressively more difficult and time consuming.…”

“…The European Association of Urology (EAU) guidelines strongly recommend the description of all macroscopic features on a bladder diagram; however, this method lacks documentation accuracy and is subject to inter-and intra-observer variability [3,5]. Recent technological alternatives have sought to improve patient surveillance and cystoscopic reproducibility through lesion positioning into 3D computerized bladder models [5,6]. Nonetheless, these methods remain to be investigated in clinical trials.…”

Reconstructing a 3D heart surface with stereo-endoscope by learning eigen-shapes

“…augmented reality guidance [7], and multispectral [8] or multimodal [9] imaging. In addition, it is useful for offline applications as well, such as surgery simulation [10] and visual medical record [11], for learning, training or evaluation purposes.However, in a highly dynamic MIS, it is very challenging to track and reconstruct Regions of Interest (ROIs) with complex soft-tissue deformations from real-time endoscopic videos. Model-based methods have been explored [12][13][14][15][16][17][18].…”

“…Model complexity (rank) can be simply measured by the number of model parameters. Low-rank models, such as rigid and affine models, are robust and computationally efficient but difficult to deal with complex soft-tissue deformations, while high-rank models generally suffer from problems of parameter convergence and heavy computational burden, difficult to meet real-time and robustness requirements, as indicated in [16].On the other hand, most existing models, typically based on the mesh [2,11,13,19] or spline [12,14,[16][17][18], are designed for general deformations with certain continuity and smoothness constraints, which do not take into account the statistics and specificities of ROIs. Statistical shape models have been explored to reconstruct static anatomical structures, such as bone surfaces from X-ray images [20] and employed as a priori information in a wide range…”