Colorectal cancer lymph node metastasis prediction with weakly supervised transformer-based multi-instance learning

Abstract

Lymph node metastasis examined by the resected lymph nodes is considered one of the most important prognostic factors for colorectal cancer (CRC). However, it requires careful and comprehensive inspection by expert pathologists. To relieve the pathologists’ burden and speed up the diagnostic process, in this paper, we develop a deep learning system with the binary positive/negative labels of the lymph nodes to solve the CRC lymph node classification task. The multi-instance learning (MIL) framework is adopted in our method to handle the whole slide images (WSIs) of gigapixels in size at once and get rid of the labor-intensive and time-consuming detailed annotations. First, a transformer-based MIL model, DT-DSMIL, is proposed in this paper based on the deformable transformer backbone and the dual-stream MIL (DSMIL) framework. The local-level image features are extracted and aggregated with the deformable transformer, and the global-level image features are obtained with the DSMIL aggregator. The final classification decision is made based on both the local and the global-level features. After the effectiveness of our proposed DT-DSMIL model is demonstrated by comparing its performance with its predecessors, a diagnostic system is developed to detect, crop, and finally identify the single lymph nodes within the slides based on the DT-DSMIL and the Faster R-CNN model. The developed diagnostic model is trained and tested on a clinically collected CRC lymph node metastasis dataset composed of 843 slides (864 metastasis lymph nodes and 1415 non-metastatic lymph nodes), achieving the accuracy of 95.3% and the area under the receiver operating characteristic curve (AUC) of 0.9762 (95% confidence interval [CI]: 0.9607–0.9891) for the single lymph node classification. As for the lymph nodes with micro-metastasis and macro-metastasis, our diagnostic system achieves the AUC of 0.9816 (95% CI: 0.9659–0.9935) and 0.9902 (95% CI: 0.9787–0.9983), respectively. Moreover, the system shows reliable diagnostic region localizing performance: the model can always identify the most likely metastases, no matter the model’s predictions or manual labels, showing great potential in avoiding false negatives and discovering incorrectly labeled slides in actual clinical use. Graphical Abstract

Computational methods for metastasis detection in lymph nodes and characterization of the metastasis-free lymph node microarchitecture: A systematic-narrative hybrid review

“…A two-step LN detection approach has also been implemented by Beuque et al, 30 where a U-Net model was trained to obtain LN mask from WSI thumbnail, followed by false-positive prediction filtering by XGBoost model trained on hand-crafted radiomics features obtained from LN masks. Faster-RCNN model was also used for LN detection by Tan et al, 31 where a model was trained to detect LN bounding boxes in colorectal cancer patients at 5x magnification.…”

“…Yu et al 82 proposed vocabulary-based MIL approach, where the model was trained to discover the structural prototypes metastatic LNs from breast cancer patients via unsupervised clustering, thus offering a higher model interpretability. Tan et al 31 proposed a transformer-based MIL model for colorectal cancer cases. The authors emphasized that the attention map obtained with the transformer model correctly localized the metastatic LN lesions even if the final prediction was negative, suggesting the model’s capability of avoiding false negatives.…”

“…Only 12 studies (19%) out of 63 studies investigating LN metastasis provided a detailed analysis of false-positive predictions in metastatic LN detection. Based on the gathered data, LN sinus histiocytes were the most common false positively identified region mentioned by six studies 29 , 31 , 56 , 68 , 90 , 94 followed by secondary lymphoid follicles (germinal centers), 56 , 64 , 94 connective tissue, 31 , 64 out-of-focus areas, 68 and slide artifacts. 50 The most common metastatic regions misclassified as negative were micrometastatic lesions 61 , 64 as well as histiocyte-like tumor cells.…”

“…In total, we found 27 distinct model architectures with 23 models describing patch classification and the remaining 4 models describing pixel-level segmentation studies. Among patch classifiers, ResNet models were utilized most frequently (15 studies), 27 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 72 , 84 , 98 followed by MIL ( n =9), 31 , 41 , 77 , 78 , 79 , 80 , 81 , 82 , 99 VGG models ( n =8), 36 , 37 , 44 , 46 , 51 , 84 , 100 , 101 and others as shown in Fig. 4 B. Amongst pixel-level classification methods, U-Net was the most commonly used (10 studies) 26 , 27 , 30 , 59 , 60 , 61 , 62 , 71 , 96 , 102 followed by DeepLabv3 ( n =7), 28 , 29 , 62 , 63 , 64 , 65 , 66 Mask-RCNN ( n =1), 16 and proprietary Visiopharm AI metastasis model 90 as can be seen in Fig.…”

A survey of Transformer applications for histopathological image analysis: New developments and future directions

“…For breast cancer histopathological image classification, DCET-Net [ 72 ] proposed a dual-stream convolution-expanded transformer architecture; Breast-Net [ 51 ] explores the ability of ensemble learning techniques using four Swin transformer architectures; HATNet [ 52 ] uses end-to-end vision transformers with a self-attention mechanism; ScoreNet [ 16 ] developed an efficient transformer-based architecture that integrates a coarse-grained global attention framework with a fine-grained local attention mechanism framework; LGVIT [ 73 ] built a local–global ViT model by introducing a new local–global MHSA mechanism and a ghost geed-forward network block into the network; dMIL-transformer [ 53 ] developed a two-stage double max–min multiple-instance learning (MIL) transformer architecture that combines both the spatial and morphological information of the cancer regions. Other than breast cancer classification, transformers have also been applied to other histopathological image cancer classification tasks, such as bone cancer classification (NRCA-FCFL [ 74 ]), brain cancer classification (ViT-WSI [ 17 ], ASI-DBNet [ 54 ], Ding et al [ 55 ]), colorectal cancer classification (MIST [ 75 ], DT-DSMIL [ 56 ]), gastric cancer classification (IMGL-VTNet [ 57 ]), kidney subtype classification (i-ViT [ 59 ], tRNAsformer [ 58 ]), thymoma or thymic carcinoma classification (MC-ViT [ 76 ]), lung cancer classification (GTP [ 46 ], FDTrans [ 60 ]), skin cancer classification (Wang et al [ 45 ]), and thyroid cancer classification (Wang et al [ 77 ], PyT2T-ViT [ 41 ], Wang et al [ 78 ]) using different transformer-based architectures. Furthermore, other transformer models such as Transmil [ 65 ], KAT [ 61 ], ViT-based unsupervised contrastive learning architecture [ 79 ], DecT [ 66 ], StoHisNet [ 80 ], CWC-transformer [ 63 ], LA-MIL [ 44 ], SETMIL [ 81 ], Prompt-MIL [ 67 ], GLAMIL [ 67 ], MaskHIT [ 82 ], HAG-MIL [ 68 ], MEGT [ 47 ], MSPT [ 70 ], and HistPathGPT [ 69 ] have also been evaluated on more than one tissue type, such as liver, prostate, breast, brain, gastric, kidney, lung, colorectal, and so on, for h...…”