Real-time image composition of bladder mosaics in fluorescence endoscopy

Abstract

Today, photodynamic diagnostics is commonly used in endoscopic intervention of the urinary bladder. Excited by a narrow band illumination, fluorescence markers enhance the visual contrast between benign and malignant tissue. Since in this modality the endoscope must be moved close to the bladder wall to provide sufficiently exposed images, the field of view (FOV) of the endoscope is very limited. This impedes the navigation and the re-identifying of multi-focal tumors for the physician. Thus, an image providing a larger FOV, composed from single images is highly desired during the intervention for surgery assistance. Since endoscopic mosaicking in real-time is still an open issue, we introduce a new feature-based image mosaicking algorithm for fluorescence endoscopy. Using a multi-threaded software design, the extraction of SURF features, the matching and the image stitching are separated in single processing threads. In an optimization step we discuss the tradeoff between feature repeatability and processing time. After adjusting an optimal thread synchronization, the optimal workload of each thread results in a fast and real-time capable computation of image mosaics. On a standard hardware platform our algorithm performs within the RealTimeFrame framework with an update rate of 8.17 frames per second at full input image resolution (780 × 576). Providing a fast A. Behrens ( ) · M. Bommes · T. Stehle · S. Gross · T. Aach growing image with an extended FOV during the intervention, the mosaic displayed on a second monitor promises high potential for surgery assistance.

Stitching and Surface Reconstruction From Endoscopic Image Sequences: A Review of Applications and Methods

“…A selection of recent results of endoscopic view expansion, presented for different fields of application. Top row: A planar panorama image, generated in real-time from fluorescence cystoscopy video frames [12]. A 3D reconstruction using shape from motion of an excavated pig bladder [134].…”

A non-linear multi-scale blending algorithm for fluorescence bladder images

“…The composition of a fluorescence image mosaic is applied by our real-time capable PDD multi-threaded image mosaicking algorithm [3,4]. Implemented in the RealTimeFrame software framework [14,15], the multi-threaded algorithm stitches subsequent images of an endoscopic video sequence into one global image composition.…”

“…It computes for each pixel the Euclidean distance from the border of a region. Applying the feathering method with similar linear distance maps on endoscopic images, seamless panoramic output images can already be provided, assuming low illumination variations across the input video sequence [3,4,26].…”

“…The size of the overlap region I A∩B is determined by the total number of NM pixels. Instead of applying a mutual crossfade interpolation [3,4,26], again the pixel values of the region mask are set equal to the illumination mask of the image with the highest mean intensity. An example of the generation of M c , where the intensity E A is higher than E B is given in Fig.…”

“…To overcome this problem, methods to provide a larger field of view of the operation field in one panoramic image, were recently published in the literature [3,4,[18][19][20].…”