Criteria2Query: a natural language interface to clinical databases for cohort definition

Yuan, Chi; Ryan, Patrick B.; Ta, Casey; Guo, Yao; Li, Ziran; Hardin, Jill; Makadia, Rupa; Peng, Jin; Shang, Ning; Kang, Tian; Weng, Chunhua

doi:10.1093/jamia/ocy178

Cited by 107 publications

(89 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recall for the structured queries varied widely across topics (Figure 3). There was 100% recall of word-based query relevant patients on 8 of the 56 topics, greater than 50% recall on 35 of the 56 topics, less then 50% recall on 13 of the 56, one topic (48) with no recall of relevant patients, and two topics with no retrieval at all (22,25).…”

Section: Structured Queriesmentioning

confidence: 99%

“…Another thread of work has focused on making querying easier to carry out, typically through development of natural language or other structured interfaces to the patient data [22][23][24][25]. Other approaches focus on normalizing semantic representation of patient data within the EHR itself [26] and applying deep learning to non-topical characteristics of studies and researchers [27].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Evaluation of Patient-Level Retrieval from Electronic Health Record Data for a Cohort Discovery Task

Bedrick

Cohen

Wang

et al. 2019

Preprint

View full text Add to dashboard Cite

Objective Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well-understood. The objective of this research was to assess patient-level information retrieval (IR) methods using electronic health records (EHR) for different types of cohort definition retrieval. Materials and Methods We developed a test collection consisting of about 100,000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated IR tasks using word-based approaches were performed, varying four different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics. Results The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision, but were still not able to recall all relevant patients found by the automated queries. Conclusion While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Insights gained in this preliminary analysis will help guide future work to develop new methods for patient-level cohort discovery with EHR data.

show abstract

Section: Structured Queriesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Evaluation of Patient-Level Retrieval from Electronic Health Record Data for a Cohort Discovery Task

Bedrick

Cohen

Wang

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Among ML approaches, Ding et al, adopt "multitask learning", and find that multitask deep learning nets outperform the simpler single-task nets-a counterintuitive observation. Yet another interesting study combined ML and rule-based approaches to identify entities and relations, providing a natural language interface to clinical databases [72].…”

Section: Phenotyping and Cohort Discoverymentioning

confidence: 99%

Review of Clinical Research Informatics

Solomonides

2020

Yearb Med Inform

View full text Add to dashboard Cite

Objectives: Clinical Research Informatics (CRI) declares its scope in its name, but its content, both in terms of the clinical research it supports—and sometimes initiates—and the methods it has developed over time, reach much further than the name suggests. The goal of this review is to celebrate the extraordinary diversity of activity and of results, not as a prize-giving pageant, but in recognition of the field, the community that both serves and is sustained by it, and of its interdisciplinarity and its international dimension. Methods: Beyond personal awareness of a range of work commensurate with the author’s own research, it is clear that, even with a thorough literature search, a comprehensive review is impossible. Moreover, the field has grown and subdivided to an extent that makes it very hard for one individual to be familiar with every branch or with more than a few branches in any depth. A literature survey was conducted that focused on informatics-related terms in the general biomedical and healthcare literature, and specific concerns (“artificial intelligence”, “data models”, “analytics”, etc.) in the biomedical informatics (BMI) literature. In addition to a selection from the results from these searches, suggestive references within them were also considered. Results: The substantive sections of the paper—Artificial Intelligence, Machine Learning, and “Big Data” Analytics; Common Data Models, Data Quality, and Standards; Phenotyping and Cohort Discovery; Privacy: Deidentification, Distributed Computation, Blockchain; Causal Inference and Real-World Evidence—provide broad coverage of these active research areas, with, no doubt, a bias towards this reviewer’s interests and preferences, landing on a number of papers that stood out in one way or another, or, alternatively, exemplified a particular line of work. Conclusions: CRI is thriving, not only in the familiar major centers of research, but more widely, throughout the world. This is not to pretend that the distribution is uniform, but to highlight the potential for this domain to play a prominent role in supporting progress in medicine, healthcare, and wellbeing everywhere. We conclude with the observation that CRI and its practitioners would make apt stewards of the new medical knowledge that their methods will bring forward.

show abstract

“…Leveraging the sizable amount of eligibility criteria information stored within CT.gov, previous work has been done to parse and map these eligibility criteria to medical concepts within the OMOP CDM (17). Using Criteria2Query's pre-parsed eligibility criteria, total inclusion and exclusion criteria counts were extracted.…”

Section: Eligibility Criteria Statisticsmentioning

confidence: 99%

“…Another limitation is that automated eligibility criteria parsing can be incomplete and occasionally inaccurate (17). In this work, many pitfalls were avoided by limiting the analysis to eligibility metadata (e.g.…”

Section: Limitations Of This Studymentioning

confidence: 99%

Finding Success in Clinical Trial Recruitment: A Trial Registry Analysis

Butler

Liu

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

Background : Randomized clinical trials are the gold-standard for generating high-quality medical evidence, but patient recruitment remains one of the most important barriers to their success despite significant administrative effort and money being spent to address this problem. While previous studies have highlighted key trial design characteristics, such as trial phase, trial sponsor, and high target accrual, as important factors in why some trials fail to recruit enough patients, these studies have been limited in the number of trials analyzed and in the scope of trial characteristics considered. This work aims to thoroughly assess the association of different trial characteristics on patient enrollment in terms of recruitment rate and early termination rate on a larger scale than has been accomplished previously. Methods : This trial registration analysis collected recruitment information on clinical trials registered in ClinicalTrials.gov as well as trial characteristics from multiple additional databases (Clinical Trials Transformation Initiative, COHD.io, automatically parsed eligibility criteria). Descriptive statistics were calculated and the primary outcomes were associations of individual trial characteristics with patient recruitment rate and likelihood of early termination due to failed patient recruitment as well as variable selection using Group LASSO. Results : The trial characteristics with the strongest significant associations to patient recruitment included design variables (e.g. intervention model, allocation status, number of locations, phase, etc. ), sponsor experience (e.g. sponsor class, number of previous trials terminated due to recruitment issues, ratio of terminated trials to completed trials, etc. ), eligibility criteria (e.g. number of inclusion criteria, number of exclusion criteria), and trial competition (e.g. overlapping eligibility criteria, similar trials within 100 miles, etc. ). Different disease categories also showed different recruitment efficacy. Conclusions : When designing clinical trials, special attention should be paid to design variables, sponsor trial experience, eligibility criteria, and trial competition to balance the likelihood of successful recruitment against evidence strength. Further research is needed to identify causal variables and improve the predictive power of patient recruitment rates to increase the breadth of this analysis.

show abstract

Criteria2Query: a natural language interface to clinical databases for cohort definition

Cited by 107 publications

References 23 publications

Evaluation of Patient-Level Retrieval from Electronic Health Record Data for a Cohort Discovery Task

Evaluation of Patient-Level Retrieval from Electronic Health Record Data for a Cohort Discovery Task

Review of Clinical Research Informatics

Finding Success in Clinical Trial Recruitment: A Trial Registry Analysis

Contact Info

Product

Resources

About