3D Registration of pre-surgical prostate MRI and histopathology images via super-resolution volume reconstruction

Sood, Rewa; Shao, Wei; Kunder, Christian A.; Teslovich, Nikola C.; Wang, Jeffrey B.; Soerensen, Simon John Christoph; Madhuripan, Nikhil; Jawahar, A.; Brooks, James D.; Ghanouni, Pejman; Fan, Richard E.; Sonn, Geoffrey A.; Rusu, Mirabela

doi:10.1016/j.media.2021.101957

Cited by 28 publications

(22 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some researchers also investigated the through-plane resolution problem by using deep learning methods for other body regions. 14,15,25,[27][28][29][30] However, most of them still require high-resolution MR images during training. 25,[27][28][29] Second, the pelvic floor has a complex structure and large variability in the shape and size of different organs.…”

Section: Discussionmentioning

confidence: 99%

Three‐dimensional self super‐resolution for pelvic floor MRI using a convolutional neural network with multi‐orientation data training

et al. 2022

View full text Add to dashboard Cite

High-resolution pelvic magnetic resonance (MR) imaging is important for the high-resolution and high-precision evaluation of pelvic floor disorders (PFDs), but the data acquisition time is long. Because high-resolution threedimensional (3D) MR data of the pelvic floor are difficult to obtain, MR images are usually obtained in three orthogonal planes: axial, sagittal, and coronal. The in-plane resolution of the MR data in each plane is high, but the through-plane resolution is low. Thus, we aimed to achieve 3D super-resolution using a convolutional neural network (CNN) approach to capture the intrinsic similarity of low-resolution 3D MR data from three orientations. Methods: We used a two-dimensional (2D) super-resolution CNN model to solve the 3D super-resolution problem. The residual-in-residual dense block network (RRDBNet) was used as our CNN backbone. For a given set of low through-plane resolution pelvic floor MR data in the axial or coronal or sagittal scan plane, we applied the RRDBNet sequentially to perform super-resolution on its two projected low-resolution views.Three datasets were used in the experiments, including two private datasets and one public dataset. In the first dataset (dataset 1), MR data acquired from 34 subjects in three planes were used to train our super-resolution model, and low-resolution MR data from nine subjects were used for testing. The second dataset (dataset 2) included a sequence of relatively high-resolution MR data acquired in the coronal plane. The public MR dataset (dataset 3) was used to demonstrate the generalization ability of our model. To show the effectiveness of RRDBNet, we used datasets 1 and 2 to compare RRDBNet with interpolation and enhanced deep super-resolution (EDSR) methods in terms of peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index. As 3D MR data from one view have two projected lowresolution views, different super-resolution orders were compared in terms of PSNR and SSIM. Finally, to demonstrate the impact of super-resolution on the image analysis task, we used datasets 2 and 3 to compare the performance of our method with interpolation on the 3D geometric model reconstruction of the urinary bladder. Results: A CNN-based method was used to learn the intrinsic similarity among MR acquisitions from different scan planes. Through-plane super-resolution for pelvic MR images was achieved without using high-resolution 3D data, which is useful for the analysis of PFDs.

show abstract

Section: Discussionmentioning

confidence: 99%

Three‐dimensional self super‐resolution for pelvic floor MRI using a convolutional neural network with multi‐orientation data training

et al. 2022

View full text Add to dashboard Cite

show abstract

“…An alternative approach to labeling cancer location on MRI is to perform automatic registration of preoperative MRI and digital histopathology images from patients undergoing radical prostatectomy. 22 , 23 , 24 , 25 , 26 , 27 Labels obtained from automatic registration are more accurate than radiologist labels since they do not depend on human interpretation of MRI and allow for the full extent of lesions found on histopathology to be mapped on MRI, including cancers that are invisible or hardly visible. Figure 1 illustrates how cancer labels mapped from histopathology images onto MRI typically extend beyond the radiologist annotation and often include cancers that were not detected by the radiologist.…”

Section: Introductionmentioning

confidence: 99%

“…Moreover, we have recently shown that deep learning methods can accelerate this registration, 23 while slice‐to‐slice correspondences are not required when using super‐resolution generative adversarial networks to reconstruct 3D histopathology and MRI volumes. 24 We previously used a subset of the unique dataset generated by RAPSODI 22 to train a deep learning model to automatically detect prostate cancer on MRI. 11 Here, we seek to expand upon this work by focusing on distinguishing aggressive from indolent cancers on MRI using labels derived from automated registration of histopathology and MR images.…”

Section: Introductionmentioning

confidence: 99%

Automated detection of aggressive and indolent prostate cancer on magnetic resonance imaging

et al. 2021

Self Cite

View full text Add to dashboard Cite

Purpose While multi‐parametric magnetic resonance imaging (MRI) shows great promise in assisting with prostate cancer diagnosis and localization, subtle differences in appearance between cancer and normal tissue lead to many false positive and false negative interpretations by radiologists. We sought to automatically detect aggressive cancer (Gleason pattern ≥ 4) and indolent cancer (Gleason pattern 3) on a per‐pixel basis on MRI to facilitate the targeting of aggressive cancer during biopsy. Methods We created the Stanford Prostate Cancer Network (SPCNet), a convolutional neural network model, trained to distinguish between aggressive cancer, indolent cancer, and normal tissue on MRI. Ground truth cancer labels were obtained by registering MRI with whole‐mount digital histopathology images from patients who underwent radical prostatectomy. Before registration, these histopathology images were automatically annotated to show Gleason patterns on a per‐pixel basis. The model was trained on data from 78 patients who underwent radical prostatectomy and 24 patients without prostate cancer. The model was evaluated on a pixel and lesion level in 322 patients, including six patients with normal MRI and no cancer, 23 patients who underwent radical prostatectomy, and 293 patients who underwent biopsy. Moreover, we assessed the ability of our model to detect clinically significant cancer (lesions with an aggressive component) and compared it to the performance of radiologists. Results Our model detected clinically significant lesions with an area under the receiver operator characteristics curve of 0.75 for radical prostatectomy patients and 0.80 for biopsy patients. Moreover, the model detected up to 18% of lesions missed by radiologists, and overall had a sensitivity and specificity that approached that of radiologists in detecting clinically significant cancer. Conclusions Our SPCNet model accurately detected aggressive prostate cancer. Its performance approached that of radiologists, and it helped identify lesions otherwise missed by radiologists. Our model has the potential to assist physicians in specifically targeting the aggressive component of prostate cancers during biopsy or focal treatment.

show abstract

“…Other researchers have also studied the approach to predict HR MR image using inputs of multiple-sequence MR images. [20][21][22] The advantage of this tactic is that it could incorporate information from multiple MR sequences. As such, if one structure information is lost at one location in the interested MR sequence, it could be captured by the other MR sequence.…”

Section: Introductionmentioning

confidence: 99%

Synthesizing high‐resolution magnetic resonance imaging using parallel cycle‐consistent generative adversarial networks for fast magnetic resonance imaging

et al. 2021

View full text Add to dashboard Cite

The common practice in acquiring the magnetic resonance (MR) images is to obtain two-dimensional (2D) slices at coarse locations while keeping the high in-plane resolution in order to ensure enough body coverage while shortening the MR scan time. The aim of this study is to propose a novel method to generate HR MR images from low-resolution MR images along the longitudinal direction. In order to address the difficulty of collecting paired low-and highresolution MR images in clinical settings and to gain the advantage of parallel cycle consistent generative adversarial networks (CycleGANs) in synthesizing realistic medical images, we developed a parallel CycleGANs based method using a self -supervised strategy. Methods and materials:The proposed workflow consists of two parallely trained CycleGANs to independently predict the HR MR images in the two planes along the directions that are orthogonal to the longitudinal MR scan direction. Then, the final synthetic HR MR images are generated by fusing the two predicted images. MR images, including T1-weighted (T1), contrast enhanced T1-weighted (T1CE), T2-weighted (T2), and T2 Fluid Attenuated Inversion Recovery (FLAIR), of the multimodal brain tumor segmentation challenge 2020 (BraTS2020) dataset were processed to evaluate the proposed workflow along the cranial-caudal (CC), lateral, and anterior-posterior directions. Institutional collected MR images were also processed for evaluation of the proposed method. The performance of the proposed method was investigated via both qualitative and quantitative evaluations. Metrics of normalized mean absolute error (NMAE), peak signal-to-noise ratio (PSNR), edge keeping index (EKI), structural similarity index measurement (SSIM), information fidelity criterion (IFC), and visual information fidelity in pixel domain (VIFP) were calculated. Results: It is shown that the proposed method can generate HR MR images visually indistinguishable from the ground truth in the investigations on the BraTS2020 dataset. In addition, the intensity profiles, difference images and SSIM maps can also confirm the feasibility of the proposed method for synthesizing HR MR images. Quantitative evaluations on the BraTS2020 dataset shows that the calculated metrics of synthetic HR MR images can all be enhanced for the T1, T1CE, T2, and FLAIR images. The enhancements in the numerical metrics over the low-resolution and bi-cubic interpolated MR images, as well as those genearted with a comparative deep learning method, are statistically significant. Qualitative evaluation of the synthetic HR MR images of

show abstract

3D Registration of pre-surgical prostate MRI and histopathology images via super-resolution volume reconstruction

Cited by 28 publications

References 34 publications

Three‐dimensional self super‐resolution for pelvic floor MRI using a convolutional neural network with multi‐orientation data training

Three‐dimensional self super‐resolution for pelvic floor MRI using a convolutional neural network with multi‐orientation data training

Automated detection of aggressive and indolent prostate cancer on magnetic resonance imaging

Synthesizing high‐resolution magnetic resonance imaging using parallel cycle‐consistent generative adversarial networks for fast magnetic resonance imaging

Contact Info

Product

Resources

About