A multimodal deep architecture for traditional Chinese medicine diagnosis

Abstract

Traditional Chinese medicine (TCM) is found on a long‐term medical practice in China. Rare human brains can fully grasp the deep TCM knowledge derived from a tremendous amount of experience. In this big data era, a big electronic brain might be competent via deep learning techniques. For this prospect, the electronic brain needs to process various heterogeneous data, such as images, texts, audio signals, and other sensory data. It used to be a challenge to analyze the heterogeneous data by the computer‐aided system until the advances of the powerful deep learning tools. We propose a multimodal deep learning framework to mimic a TCM practitioner to diagnose a patient on the basis of multimodal perceptions of see, listen, smell, ask, and touch. The framework learns common representations from various high‐dimensional sensory data, and fuse the information for final classification. We propose to use conceptual alignment deep neural networks to embed prior knowledge and obtain interpretable latent representations. We implement a multimodal deep architecture to process tongue image and description text data for TCM diagnosis. Experiments illustrate that the multimodal deep architecture can extract effective features from heterogeneous data, produce interpretable representations, and finally achieve a higher accuracy than either corresponding unimodal architectures.

Knowledge-Based Recurrent Neural Network for TCM Cerebral Palsy Diagnosis

“…Let w t be the word embedding of s t , u t be the 768dim word embedding generated by using bert-base-chinese, and v t be the embedding of s t in E r . e BERTembedding can be integrated by equation (11). Here, β ∈ [0, 1] is a hyperparameter, and G ∈ R D×r is a trainable matrix.…”

“…Yang et al [10] developed a novel deep neural network that uses multiview features of the gene data to identify the disease genes. Dai et al [11] proposed a multimodal deep learning framework based on the four-diagnosis of TCM.…”

“…Yang et al [ 10 ] developed a novel deep neural network that uses multiview features of the gene data to identify the disease genes. Dai et al [ 11 ] proposed a multimodal deep learning framework based on the four-diagnosis of TCM. These approaches effectively compensate for the information in a single-modal and improve the accuracy of the model.…”

Design and Implementation of the Traditional Chinese Medicine Constitution System Based on the Diagnosis of Tongue and Consultation

“…2) of the human face and their coordinate values can be extracted by using Dlib related APIs. This paper considers the (6,48,64,10) point area as the tongue area, which is convenient for the subsequent analysis. Tongue recognition is achieved by extracting edge features and using these features to train the SVM classifier.…”

“…Constitution identification, also known as the "four diagnostics of TCM," which traditionally involve looking, hearing, and consultation, is a crucial process in the practice of TCM [2]. Constitution identification can be applied not only to patients [3][4][5] but also to the general population to help people better understand their health status to prevent disease [6]. This paper will focus on the relationship between tongue diagnosis, consultation and constitution.…”