Identification and Evaluation of Controlled Trials in Pediatric Cardiology: Crowdsourced Scoping Review and Creation of Accessible Searchable Database

Ashkanase, Jenna; Nama, Nassr; Sandarage, Ryan; Penslar, Joshua; Gupta, Ram N.; Ly, Sophia; Wan, Melissa; Tsang, Phillip; Nantsios, Alex; Jacques, Eric; Yang, Hsin Yun; Tsang, Conrad I.; Mazhar, Hajra; Xu, Gang; Rodriguez, Maria Lorena; Gerber, Samantha; Laird, Laurie M.; Sampson, Margaret; Wong, Derek; McNally, James Dayre

doi:10.1016/j.cjca.2020.01.028

Cited by 14 publications

(20 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All six systematic reviews used in the Crowd Screening pilot met inclusion criteria. These reviews covered the areas of cardiology (12), anesthesiology, endocrinology (13), patient education, general surgery (14), and respirology (15). The characteristics of the reviews are presented in Table 1.…”

Section: Resultsmentioning

confidence: 99%

Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

Sampson

Nama

O’Hearn

et al. 2020

Int J Technol Assess Health Care

Self Cite

View full text Add to dashboard Cite

Introduction Solutions like crowd screening and machine learning can assist systematic reviewers with heavy screening burdens but require training sets containing a mix of eligible and ineligible studies. This study explores using PubMed's Best Match algorithm to create small training sets containing at least five relevant studies. Methods Six systematic reviews were examined retrospectively. MEDLINE searches were converted and run in PubMed. The ranking of included studies was studied under both Best Match and Most Recent sort conditions. Results Retrieval sizes for the systematic reviews ranged from 151 to 5,406 records and the numbers of relevant records ranged from 8 to 763. The median ranking of relevant records was higher in Best Match for all six reviews, when compared with Most Recent sort. Best Match placed a total of thirty relevant records in the first fifty, at least one for each systematic review. Most Recent sorting placed only ten relevant records in the first fifty. Best Match sorting outperformed Most Recent in all cases and placed five or more relevant records in the first fifty in three of six cases. Discussion Using a predetermined set size such as fifty may not provide enough true positives for an effective systematic review training set. However, screening PubMed records ranked by Best Match and continuing until the desired number of true positives are identified is efficient and effective. Conclusions The Best Match sort in PubMed improves the ranking and increases the proportion of relevant records in the first fifty records relative to sorting by recency.

show abstract

Section: Resultsmentioning

confidence: 99%

Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

Sampson

Nama

O’Hearn

et al. 2020

Int J Technol Assess Health Care

Self Cite

View full text Add to dashboard Cite

show abstract

“…In November 2020, we exceeded 4.5 million classifications. Although to our knowledge Cochrane Crowd is the largest crowdsourcing initiative linked to evidence synthesis, several smaller research studies have also evaluated crowdsourcing for study identification [20,22,23,26,33] plus other review production tasks, such as critical appraisal [21,25,34]. These studies all show the potential of crowdsourcing to support these tasks.…”

Section: Discussionmentioning

confidence: 99%

An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials

Noel-Storr

Dooley²,

Elliott

et al. 2021

Journal of Clinical Epidemiology

View full text Add to dashboard Cite

Background and Objectives: Filtering the deluge of new research to facilitate evidence synthesis has proven to be unmanageable using current paradigms of search and retrieval. Crowdsourcing, a way of harnessing the collective effort of a ''crowd'' of people, has the potential to support evidence synthesis by addressing this information overload created by the exponential growth in primary research outputs. Cochrane Crowd, Cochrane's citizen science platform, offers a range of tasks aimed at identifying studies related to health care. Accompanying each task are brief, interactive training modules, and agreement algorithms that help ensure accurate collective decisionmaking.The aims of the study were to evaluate the performance of Cochrane Crowd in terms of its accuracy, capacity, and autonomy and to examine contributor engagement across three tasks aimed at identifying randomized trials.Study Design and Setting: Crowd accuracy was evaluated by measuring the sensitivity and specificity of crowd screening decisions on a sample of titles and abstracts, compared with ''quasi gold-standard'' decisions about the same records using the conventional methods of dual screening. Crowd capacity, in the form of output volume, was evaluated by measuring the number of records processed by the crowd, compared with baseline. Crowd autonomy, the capability of the crowd to produce accurate collectively derived decisions without the need for expert resolution, was measured by the proportion of records that needed resolving by an expert.Results: The Cochrane Crowd community currently has 18,897 contributors from 163 countries. Collectively, the Crowd has processed 1,021,227 records, helping to identify 178,437 reports of randomized controlled trials (RCTs) for Cochrane's Central Register of Controlled Trials. The sensitivity for each task was 99.1% for the RCT identification task (RCT ID), 99.7% for the RCT identification task of trials

show abstract

“…Given the large number of citations identified in the final search strategy, we will use a hybrid approach comprising crowdsourcing and a machine learning (ML) algorithm to expedite the screening of records. The crowdsourcing methodology for systematic reviews has been previously validated [19,20] and used in a variety of health research reviews to accelerate the citation screening and provide more timely research output, while still allowing for rigorous review conduct [21][22][23]. We will recruit a curated crowd of approximately 30 English-and French-speaking reviewers with content and methodological expertise from international PICU networks (eg, Canadian Critical Care Trials Group, Pediatric Acute Lung Injury, and Sepsis Investigators group), email, social media (using the hashtags #PedsICU, #PICSp, and #CCI), and a dedicated study crowdsourcing event page on insightScope [24].…”

Section: Crowdsourcingmentioning

confidence: 99%

Pediatric Chronic Critical Illness: Protocol for a Scoping Review

Zorko¹,

McNally²,

Rochwerg³

et al. 2021

JMIR Res Protoc

View full text Add to dashboard Cite

Background Improvements in the delivery of intensive care have increased survival among even the most critically ill children, thereby leading to a growing number of children with chronic complex medical conditions in the pediatric intensive care unit (PICU). Some of these children are at a significant risk of recurrent and prolonged critical illness, with higher morbidity and mortality, making them a unique population described as having chronic critical illness (CCI). To date, pediatric CCI has been understudied and lacks an accepted consensus case definition. Objective This study aims to describe the protocol and methodology used to perform a scoping review that will describe how pediatric CCI has been defined in the literature, including the concept of prolonged PICU admission and the methodologies used to develop any existing definitions. It also aims to describe patient characteristics and outcomes evaluated in the included studies. Methods We will search four electronic databases for studies that evaluated children admitted to any PICU identified with CCI. We will also search for studies describing prolonged PICU admission, as this concept is related to pediatric CCI. Furthermore, we will develop a hybrid crowdsourcing and machine learning (ML) methodology to complete citation screening. Screening and data abstraction will be performed by 2 reviewers independently and in duplicate. Data abstraction will include the details of population definitions, demographic and clinical characteristics of children with CCI, and evaluated outcomes. Results The database search, crowd reviewer recruitment, and ML algorithm development began in March 2021. Citation screening and data abstraction were completed in April 2021. Final data verification is ongoing, with analysis and results anticipated to be completed by fall 2021. Conclusions This scoping review will describe the existing or suggested definitions of pediatric CCI and important demographic and clinical characteristics of patients to whom these definitions have been applied. This review’s results will help inform the development of a consensus case definition for pediatric CCI and set a priority agenda for future research. We will use and demonstrate the validity of crowdsourcing and ML methodologies for improving the efficiency of large scoping reviews. International Registered Report Identifier (IRRID) DERR1-10.2196/30582

show abstract

Identification and Evaluation of Controlled Trials in Pediatric Cardiology: Crowdsourced Scoping Review and Creation of Accessible Searchable Database

Cited by 14 publications

References 33 publications

Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

Creating enriched training sets of eligible studies for large systematic reviews: the utility of PubMed's Best Match algorithm

An evaluation of Cochrane Crowd found that crowdsourcing produced accurate results in identifying randomized trials

Pediatric Chronic Critical Illness: Protocol for a Scoping Review

Contact Info

Product

Resources

About