Bioinspired color-near infrared endoscopic imaging system for molecular guided cancer surgery

George, Mebin; Lew, Benjamin; Blair, Steven; Zhu, Zhongmin; Liang, Zhongshuai; Srivastava, Indrajit; Chang, Austin; Choi, Hyungsoo; Kim, Kyekyoon; Nie, Shuming; Singhal, Sunil; Gruev, Viktor

doi:10.1117/1.jbo.28.5.056002

Cited by 1 publication

(2 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Drawing inspiration from the 1975 introduction of the Bayer color filter array, our group has developed pixelated color–NIR filters optimized for NIR fluorescent imaging and seamlessly integrated with an imaging device. 20 , 29 , 30 This innovation addresses coregistration issues and improves power and data efficiency. To enhance the imaging of multiple NIR fluorescent markers, we combined vertically stacked imaging technology with pixelated spectral filters, enabling the simultaneous capture of color images and three distinct observations in the NIR spectrum.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Convolutional neural network advances in demosaicing for fluorescent cancer imaging with color–near-infrared sensors

Jin,

Kondov,

Kondov

et al. 2024

J. Biomed. Opt.

Self Cite

View full text Add to dashboard Cite

. Significance Single-chip imaging devices featuring vertically stacked photodiodes and pixelated spectral filters are advancing multi-dye imaging methods for cancer surgeries, though this innovation comes with a compromise in spatial resolution. To mitigate this drawback, we developed a deep convolutional neural network (CNN) aimed at demosaicing the color and near-infrared (NIR) channels, with its performance validated on both pre-clinical and clinical datasets. Aim We introduce an optimized deep CNN designed for demosaicing both color and NIR images obtained using a hexachromatic imaging sensor. Approach A residual CNN was fine-tuned and trained on a dataset of color images and subsequently assessed on a series of dual-channel, color, and NIR images to demonstrate its enhanced performance compared with traditional bilinear interpolation. Results Our optimized CNN for demosaicing color and NIR images achieves a reduction in the mean square error by 37% for color and 40% for NIR, respectively, and enhances the structural dissimilarity index by 37% across both imaging modalities in pre-clinical data. In clinical datasets, the network improves the mean square error by 35% in color images and 42% in NIR images while enhancing the structural dissimilarity index by 39% in both imaging modalities. Conclusions We showcase enhancements in image resolution for both color and NIR modalities through the use of an optimized CNN tailored for a hexachromatic image sensor. With the ongoing advancements in graphics card computational power, our approach delivers significant improvements in resolution that are feasible for real-time execution in surgical environments.

show abstract

Section: Introductionmentioning

confidence: 99%