Evaluating the Viability of a Smartphone-Based Annotation Tool for Faster and Accurate Image Labelling for Artificial Intelligence in Diabetic Retinopathy

Abstract

Introduction Deep Learning (DL) and Artificial Intelligence (AI) have become widespread due to the advanced technologies and availability of digital data. Supervised learning algorithms have shown human-level performance or even better and are better feature extractor-quantifier than unsupervised learning algorithms. To get huge dataset with good quality control, there is a need of an annotation tool with a customizable feature set. This paper evaluates the viability of having an in house annotation tool which works on a smartphone and can be used in a healthcare setting. Methods We developed a smartphone-based grading system to help researchers in grading multiple retinal fundi. The process consisted of designing the flow of user interface (UI) keeping in view feedback from experts. Quantitative and qualitative analysis of change in speed of a grader over time and feature usage statistics was done. The dataset size was approximately 16,000 images with adjudicated labels by a minimum of 2 doctors. Results for an AI model trained on the images graded using this tool and its validation over some public datasets were prepared. Results We created a DL model and analysed its performance for a binary referrable DR Classification task, whether a retinal image has Referrable DR or not. A total of 32 doctors used the tool for minimum of 20 images each. Data analytics suggested significant portability and flexibility of the tool. Grader variability for images was in favour of agreement on images annotated. Number of images used to assess agreement is 550. Mean of 75.9% was seen in agreement. Conclusion Our aim was to make Annotation of Medical imaging easier and to minimize time taken for annotations without quality degradation. The user feedback and feature usage statistics confirm our hypotheses of incorporation of brightness and contrast variations, green channels and zooming add-ons in correlation to certain disease types. Simulation of multiple review cycles and establishing quality control can boost the accuracy of AI models even further. Although our study aims at developing an annotation tool for diagnosing and classifying diabetic retinopathy fundus images but same concept can be used for fundus images of other ocular diseases as well as other streams of medical science such as radiology where image-based diagnostic applications are utilised.

Advances in Teleophthalmology Screening for Diabetic Retinopathy

“…When combined with portable retinal imaging, artificial intelligence had diagnostic accuracy comparable to ophthalmologists. [72][73][74] Smartphone-based widefield fundus imaging has also been developed, 75 which may prove superior to other portable fundus imaging techniques, but comparative studies are needed to demonstrate its diagnostic performance and cost-effectiveness.…”

“…Artificial intelligence-based image analysis tools may also have a role, as resource-limited settings—where expert graders may be in short supply—are likely to benefit most from portable imaging. When combined with portable retinal imaging, artificial intelligence had diagnostic accuracy comparable to ophthalmologists 72–74 . Smartphone-based widefield fundus imaging has also been developed, 75 which may prove superior to other portable fundus imaging techniques, but comparative studies are needed to demonstrate its diagnostic performance and cost-effectiveness.…”

Everything real about unreal artificial intelligence in diabetic retinopathy and in ocular pathologies

“…Morya et al [ 18 ] evaluated the first smartphone-based online annotation in the world, a tool for rapid and accurate image labeling, using AI-based DL for DR. This DL model evaluated its accuracy based on a binary referral DR classification system, depending on whether a retinal image had referral DR or not.…”

Active Learning Performance in Labeling Radiology Images Is 90% Effective

“…If we could determine the most informative images and label only them, we might be able to train an intermediate AI model that would help us label the remaining images. We note in passing that while creating a (segmentation) label takes minutes, checking an already existing label for correctness only takes 10–20 s ( 9 – 11 ). If most of the labels proposed by this intermediate AI model are correct, the human workforce would be able to check these automatically-labeled images at a much faster rate than labeling them directly.…”

“…The drawing must be accurate and so this activity consumes time. Providing a single image with segmentation labeling can easily require 2-15 min in simple cases and longer for more complex cases, per image and per person (8)(9)(10)(11). To avoid human bias, each image is typically labeled by several human labelers with three labelers being a typical number (12,13).…”