Artificial Intelligence-Based Differential Diagnosis: Development and Validation of a Probabilistic Model to Address Lack of Large-Scale Clinical Datasets

Abstract

Background Machine-learning or deep-learning algorithms for clinical diagnosis are inherently dependent on the availability of large-scale clinical datasets. Lack of such datasets and inherent problems such as overfitting often necessitate the development of innovative solutions. Probabilistic modeling closely mimics the rationale behind clinical diagnosis and represents a unique solution. Objective The aim of this study was to develop and validate a probabilistic model for differential diagnosis in different medical domains. Methods Numerical values of symptom-disease associations were utilized to mathematically represent medical domain knowledge. These values served as the core engine for the probabilistic model. For the given set of symptoms, the model was utilized to produce a ranked list of differential diagnoses, which was compared to the differential diagnosis constructed by a physician in a consult. Practicing medical specialists were integral in the development and validation of this model. Clinical vignettes (patient case studies) were utilized to compare the accuracy of doctors and the model against the assumed gold standard. The accuracy analysis was carried out over the following metrics: top 3 accuracy, precision, and recall. Results The model demonstrated a statistically significant improvement (P=.002) in diagnostic accuracy (85%) as compared to the doctors’ performance (67%). This advantage was retained across all three categories of clinical vignettes: 100% vs 82% (P<.001) for highly specific disease presentation, 83% vs 65% for moderately specific disease presentation (P=.005), and 72% vs 49% (P<.001) for nonspecific disease presentation. The model performed slightly better than the doctors’ average in precision (62% vs 60%, P=.43) but there was no improvement with respect to recall (53% vs 56%, P=.27). However, neither difference was statistically significant. Conclusions The present study demonstrates a drastic improvement over previously reported results that can be attributed to the development of a stable probabilistic framework utilizing symptom-disease associations to mathematically represent medical domain knowledge. The current iteration relies on static, manually curated values for calculating the degree of association. Shifting to real-world data–derived values represents the next step in model development.

Ontology of anomalous processes diagnosis

“…8 There is a simplification of the dynamic aspect through the reduction of temporal features to the characteristic "duration" of sign (e.g., of fever). 9 Sometimes, for manifestations and diagnoses in the KB a step-by-step development of each manifestation (symptom) is represented, corresponding to one known deviation, but with an already defined time step (the segment is divided into N equal intervals). 10 Sometimes time constraints are introduced for pairwise nonintersecting intervals, which allows one to determine the allowed distance between the corresponding time instants using expressions of temporal logic.…”

“…When adapting an ontology (from universal, problem-oriented to concretized in a domain) we have the opportunity to offer experts a more appropriate terminology, in particular, in medicine to "cover" the clinical picture and the development process not only the symptoms fixed at the doctor's appointment, as in most DSS, 9,20,23,24 but alsophysical examination and relevant investigations.…”

“…Diagnosis based on machine learning methods become ineffective in domains with a high variety of processes (in biology, medicine), for example, when faced with a nonspecific manifestation of the disease ("highly effective models fail in "rare" cases, such as patients with nonspecific manifestations of the disease"). 9 In addition to causal relationships, an experts take note of (a) associative (correlating) relationships-the presence of one affects the expectation of the presence or absence of the other (the reason for the correlation or association is not defined); (b) grouping connections (the presence of two manifestations simultaneously in a patient-specific model represents a certain situation, leads one to think about a template "syndrome"). 11 Let us consider the types of statements (sentences, formulas) that establish the correspondence between the values of causes and the values of consequences in causal patterns used in diagnosis for objects (systems) with examples from different domains.…”

Ruling In and Ruling Out COVID-19: Computing SARS-CoV-2 Infection Risk From Symptoms, Imaging and Test Data

“…We assumed all of these patients presented for care in area with a local disease prevalence equivalent to the national disease prevalence of 11.1%. For patient 1 Building on work by Chishti et al, 15 we chose probabilistic models in consideration of the scarcity of detailed, individual patient data and to take advantage of the depth of published literature on aggregate symptom probabilities. Our approaches to making step-wise diagnostic assessments with incremental information mimic clinical workflows and reflect the need for transparency and accommodation of new information critical to clinical decision-making.…”

“…Addressing these issues, Chishti et al demonstrated the advantages of using flexible probabilistic frameworks built without large-scale clinical datasets to generate ranked differential diagnoses that are more accurate that those developed by physicians. 15 Combining the approaches of this prior work suggests that an appropriate diagnostic support model should rely on easily obtained symptom data, probabilistic frameworks to avoid the need for large-scale datasets, and most importantly, a flexible schema to refine predictions based on provider judgment and the ability to adapt to changes in local prevalence and current diagnostic test performance. To this end, we present a comparison and clinical validation of three such quantitative models as well as an ensemble approach to the diagnosis of COVID-19 in ambulatory and acute care settings.…”