Development of attention-based robust deep learning model for tongue diagnosis by smartphone

Abstract

Compared with tongue diagnosis using tongue image analyzers, tongue diagnosis by smartphone has great advantages in convenience and cost for universal health monitoring, but its accuracy is affected by the shooting conditions of smartphones. Developing deep learning models with high accuracy and robustness to changes in the shooting environment for tongue diagnosis by smartphone and determining the influence of environmental changes on accuracy are necessary. In our study, a dataset of 9003 images was constructed after image pre-processing and labeling. Next, we developed an attention-based deep learning model (Deep Tongue) for 8 subtasks of tongue diagnosis, including the spotted tongue, teeth-marked tongue, and fissure tongue et al, which the average AUC of was 0.90, higher than the baseline model (ResNet50) by 0.10. Finally, we analyzed the objective reasons, the brightness of the environment and the hue of images, affecting the accuracy of tongue diagnosis by smartphone through a consistency experiment of direct subject inspection and tongue image inspection. Finally, we determined the influence of environmental changes on accuracy to quantify the robustness of the Deep Tongue model through simulation experiments. Overall, the Deep Tongue model achieved a higher and more stable classification accuracy of seven tongue diagnosis tasks in the complex shooting environment of the smartphone, and the classification of tongue coating (yellow/white) was found to be sensitive to the hue of the images and therefore unreliable without stricter shooting conditions and color correction.

Developing tongue coating status assessment using image recognition with deep learning

“…Furthermore, two models were generated from images for tongue detection and tongue coating classification. Studies reporting high accuracy in the detection of the tongue and assessment of tongue conditions using deep learning for photographs that include the tongue, validate our study findings [25][26][27][28][29].…”

“…Previous studies have used a dedicated camera for collecting images of the tongue and assessed in relation to traditional Chinese medicine, diabetes, and non-alcoholic fatty liver disease [25][26][27][28]. Additionally, a deep learning model has been developed for diagnosing tongue conditions in traditional Chinese medicine using images obtained with a smartphone camera [29]. However, studies describing the application of deep learning to digital camera images for the evaluation of tongue coating status related to oral hypofunction is not known.…”