Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study

Noel-Storr, Anna; Redmond, Patrick; Lamé, Guillaume; Liberati, Elisa Giulia; Kelly, Sarah; Miller, Lucy; Dooley, Gordon; Paterson, Andy; Burt, Jenni

doi:10.1186/s12874-021-01271-4

Cited by 11 publications

(23 citation statements)

References 29 publications

(35 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For both ML algorithms, we will determine the threshold at which the sensitivity is >95% when used in combination with a single human reviewer. This approach is consistent with the individual sensitivity of expert reviewers, as described in previous studies [ 20 , 23 , 32 , 33 ].…”

Section: Methodssupporting

confidence: 88%

Pediatric Chronic Critical Illness: Protocol for a Scoping Review

Zorko¹,

McNally²,

Rochwerg³

et al. 2021

JMIR Res Protoc

View full text Add to dashboard Cite

Background Improvements in the delivery of intensive care have increased survival among even the most critically ill children, thereby leading to a growing number of children with chronic complex medical conditions in the pediatric intensive care unit (PICU). Some of these children are at a significant risk of recurrent and prolonged critical illness, with higher morbidity and mortality, making them a unique population described as having chronic critical illness (CCI). To date, pediatric CCI has been understudied and lacks an accepted consensus case definition. Objective This study aims to describe the protocol and methodology used to perform a scoping review that will describe how pediatric CCI has been defined in the literature, including the concept of prolonged PICU admission and the methodologies used to develop any existing definitions. It also aims to describe patient characteristics and outcomes evaluated in the included studies. Methods We will search four electronic databases for studies that evaluated children admitted to any PICU identified with CCI. We will also search for studies describing prolonged PICU admission, as this concept is related to pediatric CCI. Furthermore, we will develop a hybrid crowdsourcing and machine learning (ML) methodology to complete citation screening. Screening and data abstraction will be performed by 2 reviewers independently and in duplicate. Data abstraction will include the details of population definitions, demographic and clinical characteristics of children with CCI, and evaluated outcomes. Results The database search, crowd reviewer recruitment, and ML algorithm development began in March 2021. Citation screening and data abstraction were completed in April 2021. Final data verification is ongoing, with analysis and results anticipated to be completed by fall 2021. Conclusions This scoping review will describe the existing or suggested definitions of pediatric CCI and important demographic and clinical characteristics of patients to whom these definitions have been applied. This review’s results will help inform the development of a consensus case definition for pediatric CCI and set a priority agenda for future research. We will use and demonstrate the validity of crowdsourcing and ML methodologies for improving the efficiency of large scoping reviews. International Registered Report Identifier (IRRID) DERR1-10.2196/30582

show abstract

Section: Methodssupporting

confidence: 88%

Pediatric Chronic Critical Illness: Protocol for a Scoping Review

Zorko¹,

McNally²,

Rochwerg³

et al. 2021

JMIR Res Protoc

View full text Add to dashboard Cite

show abstract

“…Despite crowd sensitivity not achieving 100% for three of the four reviews used in this evaluation study, sensitivity was comparable to other similar studies run by this and other research teams 9 , 10 , 11 , 12 and potentially more accurate than having the search results screened by a single human assessor. 24 However, it is arguable that providing a measure of sensitivity where the prevalence of included studies within each of the review datasets was very low, should be considered with caution: Review 1 had a prevalence of 0.87%, Review 2: 1.07%, Review 3: 0.53%, Review 4: 2%.…”

Section: Discussionsupporting

confidence: 70%

“…In terms of the agreement algorithm, we chose an algorithm (three consecutive agreements) that had produced high collective accuracy in other similar pilot projects. 9 , 10 Would altering the consecutive number of agreeing classifications have made a difference to collective accuracy? Starting with the accuracy of a single classification, the mean accuracy of individual contributors for each review was: 84.2% sensitivity and 82.2% specificity for Review 1; 86.6% sensitivity, 84.1% specificity for Review 2; 85.1% sensitivity, 89.9% specificity for Review 3; and 89.3% sensitivity, 90.9% specificity for Review 4.…”

Section: Discussionmentioning

confidence: 99%

Crowdsourcing the identification of studies for COVID‐19‐related Cochrane Rapid Reviews

Noel-Storr

Gartlehner

Dooley³

et al. 2022

Research Synthesis Methods

Self Cite

View full text Add to dashboard Cite

Background Utilisation of crowdsourcing within evidence synthesis has increased over the last decade. Crowdsourcing platform Cochrane Crowd has engaged a global community of 22,000 people from 170 countries. The COVID‐19 pandemic presented an opportunity to engage the community and keep up with the exponential output of COVID‐19 research. Aims To test whether a crowd could accurately assess study eligibility for reviews under time constraints. Outcome measures: time taken to complete each task, time to produce required training modules, crowd sensitivity, specificity and crowd consensus. Methods We created four crowd tasks, corresponding to four Cochrane COVID‐19 Rapid Reviews. The search results of each were uploaded and an interactive training module was developed for each task. Contributors who had participated in another COVID‐19 task were invited to participate. Each task was live for 48‐h. The final inclusion and exclusion decisions made by the core author team were used as the reference standard. Results Across all four reviews 14,299 records were screened by 101 crowd contributors. The crowd completed each screening task within 48‐h for three reviews and in 52 h for one. Sensitivity ranged from 94% to 100%. Four studies, out of a total of 109, were incorrectly rejected by the crowd. However, their absence ultimately would not have altered the conclusions of the reviews. Crowd consensus ranged from 71% to 92% across the four reviews. Conclusion Crowdsourcing can play a valuable role in study identification and offers willing contributors the opportunity to help identify COVID‐19 research for rapid evidence syntheses.

show abstract

“…We successfully incorporated crowdsourcing and ML to execute a large scoping review in an efficient timeline without hindering sensitivity. We used targeted recruitment techniques to assemble an international and multidisciplinary team with a performance (sensitivity mean [ sd ], 0.92 [0.9]) consistent with or better than other studies evaluating single and crowdsourced reviewers (22, 23, 95, 96). We used a rigorous data-driven approach to conservatively integrate a ML algorithm into citation screening, which missed none of the included studies and significantly improved review efficiency.…”

Section: Discussionmentioning

confidence: 99%

Defining Pediatric Chronic Critical Illness: A Scoping Review*

Zorko

McNally

Rochwerg

et al. 2022

Pediatric Critical Care Medicine

View full text Add to dashboard Cite

Children with chronic critical illness (CCI) are hypothesized to be a high-risk patient population with persistent multiple organ dysfunction and functional morbidities resulting in recurrent or prolonged critical care; however, it is unclear how CCI should be defined. The aim of this scoping review was to evaluate the existing literature for case definitions of pediatric CCI and case definitions of prolonged PICU admission and to explore the methodologies used to derive these definitions.DATA SOURCES: Four electronic databases (Ovid Medline, Embase, CINAHL, and Web of Science) from inception to March 3, 2021. STUDY SELECTION:We included studies that provided a specific case definition for CCI or prolonged PICU admission. Crowdsourcing was used to screen citations independently and in duplicate. A machine-learning algorithm was developed and validated using 6,284 citations assessed in duplicate by trained crowd reviewers. A hybrid of crowdsourcing and machine-learning methods was used to complete the remaining citation screening. DATA EXTRACTION:We extracted details of case definitions, study demographics, participant characteristics, and outcomes assessed.DATA SYNTHESIS: Sixty-seven studies were included. Twelve studies (18%) provided a definition for CCI that included concepts of PICU length of stay (n = 12), medical complexity or chronic conditions (n = 9), recurrent admissions (n = 9), technology dependence (n = 5), and uncertain prognosis (n = 1). Definitions were commonly referenced from another source (n = 6) or opinion-based (n = 5). The remaining 55 studies (82%) provided a definition for prolonged PICU admission, most frequently greater than or equal to 14 (n = 11) or greater than or equal to 28 days (n = 10). Most of these definitions were derived by investigator opinion (n = 24) or statistical method (n = 18). CONCLUSIONS: Pediatric CCI has been variably defined with regard to the concepts of patient complexity and chronicity of critical illness. A consensus definition is needed to advance this emerging and important area of pediatric critical care research.

show abstract

Crowdsourcing citation-screening in a mixed-studies systematic review: a feasibility study

Cited by 11 publications

References 29 publications

Pediatric Chronic Critical Illness: Protocol for a Scoping Review

Pediatric Chronic Critical Illness: Protocol for a Scoping Review

Crowdsourcing the identification of studies for COVID‐19‐related Cochrane Rapid Reviews

Defining Pediatric Chronic Critical Illness: A Scoping Review*

Contact Info

Product

Resources

About