An augmented reality microscope with real-time artificial intelligence integration for cancer diagnosis

Abstract

These authors contributed equally to this work.The brightfield microscope is instrumental in the visual examination of both biological and physical samples at sub-millimeter scales. One key clinical application has been in cancer histopathology, where the microscopic assessment of the tissue samples is used for the diagnosis and staging of cancer and thus guides clinical therapy 1 . However, the interpretation of these samples is inherently subjective, resulting in significant diagnostic variability 2,3 . Moreover, in many regions of the world, access to pathologists is severely limited due to lack of trained personnel 4 . In this regard, Artificial Intelligence (AI) based tools promise to improve the access and quality of healthcare 5-7 . However, despite significant advances in AI research, integration of these tools into real-world cancer diagnosis workflows remains challenging because of the costs of image digitization and difficulties in deploying AI solutions 8 , 9 . Here we propose a cost-effective solution to the integration of AI: the Augmented Reality Microscope (ARM). The ARM overlays AI-based information onto the current view of the sample through the optical pathway in real-time, enabling seamless integration of AI into the regular microscopy workflow. We demonstrate the utility of ARM in the detection of lymph node metastases in breast cancer and the identification of prostate cancer with a latency that supports real-time workflows. We anticipate that ARM will remove barriers towards the use of AI in microscopic analysis and thus improve the accuracy and efficiency of cancer diagnosis. This approach is applicable to other microscopy tasks and AI algorithms in the life sciences 10 and beyond 11,12 .Microscopic examination of samples is the gold standard for the diagnosis of cancer, autoimmune diseases, infectious diseases, and more. In cancer, the microscopic examination of stained tissue sections is critical for diagnosing and staging the patient's tumor, which informs treatment decisions and prognosis. In cancer, microscopy analysis faces three major challenges. As a form of image interpretation, these examinations are inherently subjective, exhibiting considerable inter-observer and intra-observer variability 2,3 . Moreover, clinical guidelines 1 and studies 13 have begun to require quantitative assessments as part of the effort towards better patient risk stratification 1 . For example, breast cancer staging requires counting mitotic cells and quantification of the tumor burden in lymph nodes by measuring the largest tumor focus. However, despite being helpful in treatment planning, quantification is laborious and error-prone. Lastly, access to disease experts can be limited in both developed and developing countries 4 , exacerbating the problem.

Development of a Deep Learning Algorithm for the Histopathologic Diagnosis and Gleason Grading of Prostate Cancer Biopsies: A Pilot Study

“…There are other potential applications of a deep learning algorithm that improves the diagnosis and Gleason grading of prostate core biopsy given its central role in the evaluation and management of prostate cancer. For instance, such an algorithm might expand access to expert pathologic diagnosis not only across the USA but globally to regions where access to high-quality health care may be limited [22]. In settings with established pathologic expertise, such a system could be used to minimize human error as part of quality assurance/improvement efforts.…”

Artificial Intelligence and Mechanistic Modeling for Clinical Decision Making in Oncology

“…29 Going further, this work led to the development of an augmentedreality microscope with real-time indication of susceptible areas. 30 Pushing the concept and using DL to "see" better in digital pathology, Kather et al trained a CNN to classify the stromal components (such as lymphocytes, adipose tissue, smooth muscle, etc.) in colorectal cancer histopathological slides, using 100,000 image patches.…”

Can pathology diagnostic services for cancer be stratified and serve global health?

“…Google's Augmented Reality Microscope is the perfect example of artificial intelligence (AI) technology designed to assist a pathologist's morphologic evaluation 26 . It comprises low‐cost components (a camera, software, and an AI display) that can be retrofitted to a standard microscope.…”