Overcoming the limitations of patch-based learning to detect cancer in whole slide images

Abstract

Whole slide images (WSIs) pose unique challenges when training deep learning models. They are very large which makes it necessary to break each image down into smaller patches for analysis, image features have to be extracted at multiple scales in order to capture both detail and context, and extreme class imbalances may exist. Significant progress has been made in the analysis of these images, thanks largely due to the availability of public annotated datasets. We postulate, however, that even if a method scores well on a challenge task, this success may not translate to good performance in a more clinically relevant workflow. Many datasets consist of image patches which may suffer from data curation bias; other datasets are only labelled at the whole slide level and the lack of annotations across an image may mask erroneous local predictions so long as the final decision is correct. In this paper, we outline the differences between patch or slide-level classification versus methods that need to localize or segment cancer accurately across the whole slide, and we experimentally verify that best practices differ in both cases. We apply a binary cancer detection network on post neoadjuvant therapy breast cancer WSIs to find the tumor bed outlining the extent of cancer, a task which requires sensitivity and precision across the whole slide. We extensively study multiple design choices and their effects on the outcome, including architectures and augmentations. We propose a negative data sampling strategy, which drastically reduces the false positive rate (25% of false positives versus 62.5%) and improves each metric pertinent to our problem, with a 53% reduction in the error of tumor extent. Our results indicate classification performances of image patches versus WSIs are inversely related when the same negative data sampling strategy is used. Specifically, injection of negatives into training data for image patch classification degrades the performance, whereas the performance is improved for slide and pixel-level WSI classification tasks. Furthermore, we find applying extensive augmentations helps more in WSI-based tasks compared to patch-level image classification.

An AI-based approach for detecting cells and microbial byproducts in low volume scanning electron microscope images of biofilms

“…In order to amplify the data volume available for training the models, each image was decomposed into multiple patches. We have used the traditional sliding windows technique (Ciga et al, 2021 ; Tsai et al, 2022 ) to split an image into smaller patches using different kernel sizes and different overlapping ratios (strides). The kernel size defines the dimensions of the patch and the stride gives the next region of the image from which the patch is extracted.…”

Simulation Palynologists for Pollinosis Prevention: A Progressive Learning of Pollen Localization and Classification for Whole Slide Images

“…We first cut pollen WSIs into numerous images with small sizes called patches. This design draws on a variety of recent ideas from [ 40 , 41 ] and leverages OpenSlide [ 42 ] and NDPItools [ 43 ]. The W and P notation is defined to distinguish between “pollen WSIs” and “image patches” that correspond to that image.…”

Seeing the random forest through the decision trees. Supporting learning health systems from histopathology with machine learning models: Challenges and opportunities

“…Largely due to computational constraints to processing complete WSIs, ML models have usually been trained using sampling methods that extract small patches obtained with one or a limited number of scanning magnifications. [38][39][40][41][42][43][44] The limitations of using these sampling strategies could be illustrated in the parable of the "blind men and an elephant". 45 As ML models' pattern recognition/prediction capabilities rely on the data contained in these patches, developing efficient methods to facilitate the extraction of all the relevant information from WSIs is of paramount importance.…”