Deep Cytometry: Deep learning with Real-time Inference in Cell Sorting and Flow Cytometry

Abstract

Deep learning has achieved spectacular performance in image and speech recognition and synthesis. It outperforms other machine learning algorithms in problems where large amounts of data are available. In the area of measurement technology, instruments based on the photonic time stretch have established record real-time measurement throughput in spectroscopy, optical coherence tomography, and imaging flow cytometry. These extreme-throughput instruments generate approximately 1 Tbit/s of continuous measurement data and have led to the discovery of rare phenomena in nonlinear and complex systems as well as new types of biomedical instruments. Owing to the abundance of data they generate, time-stretch instruments are a natural fit to deep learning classification. Previously we had shown that high-throughput label-free cell classification with high accuracy can be achieved through a combination of time-stretch microscopy, image processing and feature extraction, followed by deep learning for finding cancer cells in the blood. Such a technology holds promise for early detection of primary cancer or metastasis. Here we describe a new deep learning pipeline, which entirely avoids the slow and computationally costly signal processing and feature extraction steps by a convolutional neural network that directly operates on the measured signals. The improvement in computational efficiency enables low-latency inference and makes this pipeline suitable for cell sorting via deep learning. Our neural network takes less than a few milliseconds to classify the cells, fast enough to provide a decision to a cell sorter for real-time separation of individual target cells. We demonstrate the applicability of our new method in the classification of OT-II white blood cells and SW-480 epithelial cancer cells with more than 95% accuracy in a label-free fashion.

Perspectives on liquid biopsy for label‐free detection of “circulating tumor cells” through intelligent lab‐on‐chips

“…The integration of label‐free imaging, AI, and microfluidics is a subject of great interest in the scientific community to solve open biomedical questions 21,28‐31 . In particular, the authors of the present review strongly believe that such integration would represent a keystone for the identification of CTCs (Figure 1).…”

“…The latter refers to neural networks, that is, architectures that can operate directly on the input image (rather than its descriptors) and can learn, from a wide dataset, its most appropriate representation with several levels of abstraction 151‐154 . Deep learning approaches to image segmentation and classification have been demonstrated to be robust with a huge generalization power 15,19,20,28,29,154‐168 . Cell segmentation in microfluidic streams is obtainable using pretrained networks (eg, Mask R‐CNN and Faster R‐CNN) 151,152,167 .…”

“…Deep cytometry is a recently introduced concept, where images from a QPI photonic time‐stretching 168 system create the input signals used, during the inference stage, to discern between OT‐II hybridoma T‐lymphocytes and SW‐480 colon cancer epithelial cells with accuracy higher than 95.5% and cells/s throughput. Relying on this concept, and exploiting the very fast inference time, image‐activated automatic cell sorting has been demonstrated, which is the holy grail of deep learning‐assisted flow cytometry 29 . A deep learning‐aided flow cytometer with a cameraless design exploited light‐sheet laser illumination and the readout of photomultiplier tubes to perform image‐activated sorting of pEGFP‐GR plasmids translocated and un‐translocated HEK‐293T cells, which is important because cytoplasmic mislocalization of tumor suppressor proteins can be a marker of oncogenic mutations, for example, in the case of retinoblastoma tumor 180 .…”

Detection of Rare Objects by Flow Cytometry: Imaging, Cell Sorting, and Deep Learning Approaches

“…Different types of cells are categorized and charged with different polarity charges so that they can be separated into different collection tubes. Reprinted with permission from [351]. Copyright 2019, Nature Springer.…”

“…Li et al addressed the problem of imaging cells without fluorescent agents to avoid their impact on cell behavior [351]. The authors developed an unusual approach to the problem of computational complexity.…”