A corpus of GA4GH Phenopackets: case-level phenotyping for genomic diagnostics and discovery

Danis, Daniel; Bamshad, Michael J; Bridges, Yasemin; Cacheiro, Pilar; Carmody, Leigh C; Chong, Jessica X; Coleman, Ben; Dalgleish, Raymond; Freeman, Peter J; Graefe, Adam S L; Groza, Tudor; Jacobsen, Julius O B; Klocperk, Adam; Kusters, Maaike; Ladewig, Markus S; Marcello, Anthony J; Mattina, Teresa; Mungall, Christopher J; Munoz-Torres, Monica C; Reese, Justin T; Rehburg, Filip; Reis, Bárbara C S; Schuetz, Catharina; Smedley, Damian; Strauss, Timmy; Sundaramurthi, Jagadish Chandrabose; Thun, Sylvia; Wissink, Kyran; Wagstaff, John F; Zocche, David; Haendel, Melissa A; Robinson, Peter N

doi:10.1101/2024.05.29.24308104

Cited by 2 publications

(1 citation statement)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluated the performance of GPT-4 in differential diagnosis using 5267 computational case reports formatted as GA4GH phenopackets taken from the phenopacket-store repository (version 0.14). 15 The case reports describe 378 Mendelian and chromosomal diseases associated with 336 genes. Each phenopacket contains information derived from published case or cohort reports from a total of 726 different publications.…”

Section: Study Design and Datamentioning

confidence: 99%

Evaluation of the Diagnostic Accuracy of GPT-4 in Five Thousand Rare Disease Cases

Reese,

Chimirri,

Danis

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Large language models (LLM) have shown great promise in supporting differential diagnosis, but 23 available published studies on the diagnostic accuracy evaluated small cohorts (number of cases, 30-422, mean 104) and have evaluated LLM responses subjectively by manual curation (23/23 studies). The performance of LLMs for rare disease diagnosis has not been evaluated systematically. Here, we perform a rigorous and large-scale analysis of the performance of a GPT-4 in prioritizing candidate diagnoses, using the largest-ever cohort of rare disease patients. Our computational study used 5267 computational case reports from previously published data. Each case was formatted as a Global Alliance for Genomics and Health (GA4GH) phenopacket, in which clinical anomalies were represented as Human Phenotype Ontology (HPO) terms. We developed software to generate prompts from each phenopacket. Prompts were sent to Generative Pre-trained Transformer 4 (GPT-4), and the rank of the correct diagnosis, if present in the response, was recorded.The mean reciprocal rank of the correct diagnosis was 0.24 (with the reciprocal of the MRR corresponding to a rank of 4.2), and the correct diagnosis was placed in rank 1 in 19.2% of the cases, in the first 3 ranks in 28.6%, and in the first 10 ranks in 32.5%. Our study is the largest to be reported to date and provides a realistic estimate of the performance of GPT-4 in rare disease medicine.

show abstract

Section: Study Design and Datamentioning

confidence: 99%

Evaluation of the Diagnostic Accuracy of GPT-4 in Five Thousand Rare Disease Cases

Reese,

Chimirri,

Danis

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

Towards a standard benchmark for variant and gene prioritisation algorithms: PhEval - Phenotypic inference Evaluation framework

Bridges,

de Souza,

Cortes

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

BackgroundComputational approaches to support rare disease diagnosis are challenging to build, requiring the integration of complex data types such as ontologies, gene-to-phenotype associations, and cross-species data into variant and gene prioritisation algorithms (VGPAs). However, the performance of VGPAs has been difficult to measure and is impacted by many factors, for example, ontology structure, annotation completeness or changes to the underlying algorithm. Assertions of the capabilities of VGPAs are often not reproducible, in part because there is no standardised, empirical framework and openly available patient data to assess the efficacy of VGPAs - ultimately hindering the development of effective prioritisation tools.ResultsIn this paper, we present our benchmarking tool, PhEval, which aims to provide a standardised and empirical framework to evaluate phenotype-driven VGPAs. The inclusion of standardised test corpora and test corpus generation tools in the PhEval suite of tools allows open benchmarking and comparison of methods on standardised data sets.ConclusionsPhEval and the standardised test corpora solve the issues of patient data availability and experimental tooling configuration when benchmarking and comparing rare disease VGPAs. By providing standardised data on patient cohorts from real-world case-reports and controlling the configuration of evaluated VGPAs, PhEval enables transparent, portable, comparable and reproducible benchmarking of VGPAs. As these tools are often a key component of many rare disease diagnostic pipelines, a thorough and standardised method of assessment is essential for improving patient diagnosis and care.

show abstract

A corpus of GA4GH Phenopackets: case-level phenotyping for genomic diagnostics and discovery

Cited by 2 publications

References 32 publications

Evaluation of the Diagnostic Accuracy of GPT-4 in Five Thousand Rare Disease Cases

Evaluation of the Diagnostic Accuracy of GPT-4 in Five Thousand Rare Disease Cases

Towards a standard benchmark for variant and gene prioritisation algorithms: PhEval - Phenotypic inference Evaluation framework

Contact Info

Product

Resources

About