The automatic detection and tracking of general objects (like persons, animals or cars), text and logos in a video is crucial for many video understanding tasks, and usually real-time processing as required. We propose OmniTrack, an efficient and robust algorithm which is able to automatically detect and track objects, text as well as brand logos in realtime. It combines a powerful deep learning based object detector (YoloV3) with high-quality optical flow methods. Based on the reference YoloV3 C++ implementation, we did some important performance optimizations which will be described. The major steps in the training procedure for the combined detector for text and logo will be presented. We will describe then the OmniTrack algorithm, consisting of the phases preprocessing, feature calculation, prediction, matching and update. Several performance optimizations have been implemented there as well, like doing the object detection and optical flow calculation asynchronously. Experiments show that the proposed algorithm runs in real-time for standard definition (720x576) video on a PC with a Quadro RTX 5000 GPU.

Computational pathology is revolutionizing the field of pathology by integrating advanced computer vision and machine learning technologies into diagnostic workflows. Recently, self-supervised learning (SSL) has emerged as a promising solution to learn representations from histology patches, leveraging large volumes of unannotated whole slide images (WSI). In particular, Masked Image Modeling (MIM) showed remarkable results and robustness over purely contrastive learning methods. In this work, we explore the application of MIM to histology using iBOT, a self-supervised transformer-based framework. Through a wide range of downstream tasks over seven cancer indications, we provide recommendations on the pre-training of large models for histology data using MIM. First, we demonstrate that in-domain pre-training with iBOT outperforms both ImageNet pre-training and a model pre-trained with a purely contrastive learning objective, MoCo V2. Second, we show that Vision Transformers models (ViT), when scaled appropriately, have the capability to learn pan-cancer representations that benefit a large variety of downstream tasks. Finally, our iBOT ViT-Base model, pre-trained on more than 40 million histology images from 16 different cancer types, achieves state-of-the-art performance in most weakly-supervised WSI classification tasks compared to other SSL frameworks.

No abstract

The need for developing new biomarkers is increasing with the emergence of many targeted therapies. In this study, we used artificial intelligence (AI) to develop a multimodal model (PULS-AI) predicting the survival of solid tumor patients treated with antiangiogenic treatments. Our retrospective, multicentric study included 616 patients with 7 different cancer types: renal cell carcinoma, colorectal carcinoma, hepatocellular carcinoma, gastrointestinal carcinoma, melanoma, breast cancer, and sarcoma. A set of 196 patients was left out of the study for validation. Clinical data including patient, treatment, and cancer metadata were collected at baseline for all patients, as well as computed tomography (CT) and ultrasound (US) images. Radiologists annotated all metastases on the CT images and the visible tumor lesion on the US images. AI models were used to extract relevant features from the regions of interest on CT and US images. In addition, handcrafted features related to the tumor burden were extracted from the annotations of all lesions on CT such as the number of lesions and the tumor burden volume per organ (lungs, liver, skull, bone, other). Finally, a Cox regression model was fitted to the set of imaging features and clinical features. The annotation process led to 1147 annotated US images with lesions delineation and 4564 reviewed CTs, of which 989 were selected and fully annotated with a total of 9516 annotated lesions.The developed model reaches an average concordance index of 0.71 (0.67-0.75, 95% CI). Using a risk threshold of 50%, PULS-AI model is able to significantly isolate (log-rank test P-value < 0.001) high-risk patients from low-risk patients (respective median OS of 12 and 32 months) with a hazard ratio of 3.52 (2.35-5.28, 95% CI). The results of this study show that AI algorithms are able to extract relevant information from radiology images and to aggregate data from multiple modalities to build powerful prognostic tools. Such tools may provide assistance to oncology clinicians in therapeutic decision-making. Citation Format: Kathryn Schutte, Fabien Brulport, Sana Harguem-Zayani, Jean-Baptiste Schiratti, Ridouane Ghermi, Paul Jehanno, Alexandre Jaeger, Talal Alamri, Raphael Naccache, Leila Haddag-Miliani, Teresa Orsi, Jean-Philippe Lamarque, Isaline Hoferer, Littisha Lawrance, Baya Benatsou, Imad Bousaid, Mickael Azoulay, Antoine Verdon, François Bidault, Corinne Balleyguier, Victor Aubert, Etienne Bendjebbar, Charles Maussion, Nicolas Loiseau, Benoit Schmauch, Meriem Sefta, Gilles Wainrib, Thomas Clozel, Samy Ammari, Nathalie Lassau. PULS-AI: A multimodal artificial intelligence model to predict survival of solid tumor patients treated with antiangiogenics [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 1924.

No abstract

The objective is to find a consensus definition of intra-tumor heterogeneity of liver lesions for colorectal cancer patients on computed tomography (CT) scans and to evaluate the prognostic value of the newly defined criteria. 89 patients with colorectal cancer treated by antiangiogenics having standard pretreatment CT-scans were included. Lesions were manually segmented in a previous study. We reached a consensus definition by bringing three radiologists together to determine which criteria make a tumor heterogeneous. We asked two of the three radiologists to score lesions from one to three based on the consensus definition. A subset of 50 lesions was shared between annotators in order to check their agreement. 898 lesions were scored, which corresponds to a median of 6 lesions per patient. We associate a unique heterogeneity score to each patient defined as the difference between the number of very heterogeneous tumors and the number of heterogeneous tumors. The prognostic value of this heterogeneity score was assessed using a Cox regression. Intra-tumor heterogeneity of a single lesion is defined on a scale of one to three: homogeneous (pixels are of similar intensities), heterogeneous (pixels are of different intensities but organized as a ring), very heterogeneous (pixels are of different intensities and randomly organized). We note that the shape of the tumor is not taken into account in the heterogeneity assessment. The two radiologists agreed on the lesion heterogeneity score in 53% of cases. The patient heterogeneity score is significantly prognostic in both the univariate analysis (Kaplan-Meier estimator with median cut-off; p-value = 0.016) and the multivariate analysis (Cox model with control variables including gender, age, weight, treatment and tumor burden; p-value = 0.024). Three radiologists have reached a consensus to define intra-tumor heterogeneity of liver lesions for colorectal patients. According to this definition, intra-tumor heterogeneity was shown to be a prognostic factor. Citation Format: Ridouane Ghermi, Samy Ammari, Gabriel Garcia, Sana Harguem-zayani, Nathalie Lassau, Kathryn Schutte. Intra-tumor heterogeneity assessment of liver lesions for colorectal cancer patients on CT-scans [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2023; Part 1 (Regular and Invited Abstracts); 2023 Apr 14-19; Orlando, FL. Philadelphia (PA): AACR; Cancer Res 2023;83(7_Suppl):Abstract nr 3316.