Procedural generation of synthetic multiplex immunohistochemistry images using cell-based image compression and conditional generative adversarial networks

Herbsthofer, Laurin; Ehall, Barbara; Tomberger, Martina; Prietl, Barbara; Pieber, Thomas R.; López-García, Pablo

doi:10.1117/12.2606365

Cited by 3 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… 83 , 84 In a first use-case of murine pancreatic islets, we showed how to exploit the cell-based data representation of Cell2Grid to algorithmically create synthetic image datasets with known ground truth using a procedural algorithm for the creation of Cell2Grid images and a cGAN to turn Cell2Grid images into original fm-IHC images. 85 This will support the creation of a pipeline for knowledge extraction from CNNs 75 , 86 – 89 trained on Cell2Grid images.…”

Section: Discussionmentioning

confidence: 99%

“…As summarized by Jaume et al., 76 model explanations that operate on a pixel-level suffer from real interpretability by not taking into account that the important entities of a histologic image are the individual biological cells and their spatial distribution 83 , 84 . In a first use-case of murine pancreatic islets, we showed how to exploit the cell-based data representation of Cell2Grid to algorithmically create synthetic image datasets with known ground truth using a procedural algorithm for the creation of Cell2Grid images and a cGAN to turn Cell2Grid images into original fm-IHC images 85 . This will support the creation of a pipeline for knowledge extraction from CNNs 75 , 86 – 89 trained on Cell2Grid images.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Cell2Grid: an efficient, spatial, and convolutional neural network-ready representation of cell segmentation data

et al. 2022

Self Cite

View full text Add to dashboard Cite

Purpose: Cell segmentation algorithms are commonly used to analyze large histologic images as they facilitate interpretation, but on the other hand they complicate hypothesis-free spatial analysis. Therefore, many applications train convolutional neural networks (CNNs) on highresolution images that resolve individual cells instead, but their practical application is severely limited by computational resources. In this work, we propose and investigate an alternative spatial data representation based on cell segmentation data for direct training of CNNs.Approach: We introduce and analyze the properties of Cell2Grid, an algorithm that generates compact images from cell segmentation data by placing individual cells into a low-resolution grid and resolves possible cell conflicts. For evaluation, we present a case study on colorectal cancer relapse prediction using fluorescent multiplex immunohistochemistry images.Results: We could generate Cell2Grid images at 5-μm resolution that were 100 times smaller than the original ones. Cell features, such as phenotype counts and nearest-neighbor cell distances, remain similar to those of original cell segmentation tables (p < 0.0001). These images could be directly fed to a CNN for predicting colon cancer relapse. Our experiments showed that test set error rate was reduced by 25% compared with CNNs trained on images rescaled to 5μm with bilinear interpolation. Compared with images at 1-μm resolution (bilinear rescaling), our method reduced CNN training time by 85%. Conclusions:Cell2Grid is an efficient spatial data representation algorithm that enables the use of conventional CNNs on cell segmentation data. Its cell-based representation additionally opens a door for simplified model interpretation and synthetic image generation.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Cell2Grid: an efficient, spatial, and convolutional neural network-ready representation of cell segmentation data

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…This approach likely requires a substantial amount of training data, which may not be easily accessible and therefore limits its application. However, the simplified data structure of the feature tensor and the phenotype tensor open avenues for the generation of procedural synthetic 3D tissue data 13 . This may aid in the development, training and gauging of 3D convolutional neural networks and potential knowledge extraction pipelines.…”

Section: Discussionmentioning

confidence: 99%

Cell2Voxel: a novel, cell-based 3D tissue model from 2D multiplex tissue scans

Herbsthofer¹,

Ehall

Tomberger

et al. 2023

Medical Imaging 2023: Digital and Computational Pathology

View full text Add to dashboard Cite

For many applications in oncology, pathologists inspect 2D tissue sections of resected tumor samples to assess tumor morphology, identify infiltrating lymphocytes, and predict disease progression. While 3D tissue models obtained with confocal or light-sheet microscopes promise added value over 2D scans, data-handling and visualization of whole-slide scans at cell-resolution becomes computationally demanding and sometimes even impossible on consumer-grade hardware. To enable the simultaneous display of large tissue volumes and individual cell phenotypes with a small memory footprint, we created Cell2Voxel, a light-weight, cell-based 3D tissue model in which every cell is represented by a single voxel on a coarse, regular grid. As an alternative to confocal and light-sheet microscopy, we present a workflow including widefield imaging of multiplex 2D tissue sections, cell segmentation and phenotyping, image registration, conflict-free assignment of cell-voxels to the target grid, and the creation of the final 3D tissue model. As a showcase, we present 3D models of human colon cancer samples and murine pancreatic islets. We show that Cell2Voxel can reduce the memory footprint compared to other methods by 2-3 orders of magnitude and enables the simultaneous visualization of millions of individual cells in whole-slide tissue volumes on conventional computer hardware. While our final 3D model can be inspected using open-source software on all major operating systems, we additionally present an interactive data viewer tailored to our data format. Cell2Voxel can be created from any 2D widefield image stack, making it an inexpensive, easy-to-use alternative for large-volume 3D tissue visualization.

show abstract

“…Then, an embedded map of different cellular types (Figure 2F) with a 64-fold dimensional reduction from the raw WSI is obtained. We chose a down-sampling factor under 64 because cell images are commonly cropped to 256 × 256 pixels [33][34][35].…”

Section: Cellular Embedding Of Segmented Wsimentioning

confidence: 99%

Automated Cellular-Level Dual Global Fusion of Whole-Slide Imaging for Lung Adenocarcinoma Prognosis

Diao,

Chen,

Showkatian

et al. 2023

Cancers

View full text Add to dashboard Cite

Histopathologic whole-slide images (WSI) are generally considered the gold standard for cancer diagnosis and prognosis. Survival prediction based on WSI has recently attracted substantial attention. Nevertheless, it remains a central challenge owing to the inherent difficulties of predicting patient prognosis and effectively extracting informative survival-specific representations from WSI with highly compounded gigapixels. In this study, we present a fully automated cellular-level dual global fusion pipeline for survival prediction. Specifically, the proposed method first describes the composition of different cell populations on WSI. Then, it generates dimension-reduced WSI-embedded maps, allowing for efficient investigation of the tumor microenvironment. In addition, we introduce a novel dual global fusion network to incorporate global and inter-patch features of cell distribution, which enables the sufficient fusion of different types and locations of cells. We further validate the proposed pipeline using The Cancer Genome Atlas lung adenocarcinoma dataset. Our model achieves a C-index of 0.675 (±0.05) in the five-fold cross-validation setting and surpasses comparable methods. Further, we extensively analyze embedded map features and survival probabilities. These experimental results manifest the potential of our proposed pipeline for applications using WSI in lung adenocarcinoma and other malignancies.

show abstract

Procedural generation of synthetic multiplex immunohistochemistry images using cell-based image compression and conditional generative adversarial networks

Cited by 3 publications

References 8 publications

Cell2Grid: an efficient, spatial, and convolutional neural network-ready representation of cell segmentation data

Cell2Grid: an efficient, spatial, and convolutional neural network-ready representation of cell segmentation data

Cell2Voxel: a novel, cell-based 3D tissue model from 2D multiplex tissue scans

Automated Cellular-Level Dual Global Fusion of Whole-Slide Imaging for Lung Adenocarcinoma Prognosis

Contact Info

Product

Resources

About