A pilot validation study of crowdsourcing systematic reviews: update of a searchable database of pediatric clinical trials of high-dose vitamin D

Nama, Nassr; Iliriani, Klevis; Xia, Meng; Chen, Brian P.; Zhou, Linghong Linda; Pojsupap, Supichaya; Kappel, Coralea; O’Hearn, Katie; Sampson, Margaret; Menon, Kusum; McNally, James Dayre

doi:10.21037/tp.2016.12.01

Cited by 22 publications

(19 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The primary outcome for the feasibility component was the number of citations that achieved the target number of independent assessments. Consistent with our initial pilot study, feasibility success was a priori defined as achieving a minimum of 4 independent assessments per citation [23]. The primary outcome for the validation component was the ability of the crowd to identify and retain eligible studies at the abstract level (sensitivity).…”

Section: Methodsmentioning

confidence: 99%

Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study

Nama¹,

Sampson²,

Barrowman³

et al. 2019

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

Background Systematic reviews (SRs) are often cited as the highest level of evidence available as they involve the identification and synthesis of published studies on a topic. Unfortunately, it is increasingly challenging for small teams to complete SR procedures in a reasonable time period, given the exponential rise in the volume of primary literature. Crowdsourcing has been postulated as a potential solution. Objective The feasibility objective of this study was to determine whether a crowd would be willing to perform and complete abstract and full text screening. The validation objective was to assess the quality of the crowd’s work, including retention of eligible citations (sensitivity) and work performed for the investigative team, defined as the percentage of citations excluded by the crowd. Methods We performed a prospective study evaluating crowdsourcing essential components of an SR, including abstract screening, document retrieval, and full text assessment. Using CrowdScreenSR citation screening software, 2323 articles from 6 SRs were available to an online crowd. Citations excluded by less than or equal to 75% of the crowd were moved forward for full text assessment. For the validation component, performance of the crowd was compared with citation review through the accepted, gold standard, trained expert approach. Results Of 312 potential crowd members, 117 (37.5%) commenced abstract screening and 71 (22.8%) completed the minimum requirement of 50 citation assessments. The majority of participants were undergraduate or medical students (192/312, 61.5%). The crowd screened 16,988 abstracts (median: 8 per citation; interquartile range [IQR] 7-8), and all citations achieved the minimum of 4 assessments after a median of 42 days (IQR 26-67). Crowd members retrieved 83.5% (774/927) of the articles that progressed to the full text phase. A total of 7604 full text assessments were completed (median: 7 per citation; IQR 3-11). Citations from all but 1 review achieved the minimum of 4 assessments after a median of 36 days (IQR 24-70), with 1 review remaining incomplete after 3 months. When complete crowd member agreement at both levels was required for exclusion, sensitivity was 100% (95% CI 97.9-100) and work performed was calculated at 68.3% (95% CI 66.4-70.1). Using the predefined alternative 75% exclusion threshold, sensitivity remained 100% and work performed increased to 72.9% (95% CI 71.0-74.6; P <.001). Finally, when a simple majority threshold was considered, sensitivity decreased marginally to 98.9% (95% CI 96.0-99.7; P =.25) and work performed increased substantially to 80.4% (95% CI 78.7-82.0; P <.001). Conclusions Crowdsourcing of citation screening for SRs is feasible and has reasonable sensitivity and specificity. By expediting the screening process, crowdsourcing could permit the i...

show abstract

Section: Methodsmentioning

confidence: 99%

Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study

Nama¹,

Sampson²,

Barrowman³

et al. 2019

J Med Internet Res

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition, we hand-searched: (1) references of included studies and relevant review articles; (2) conference abstracts for major gastroenterology meetings (Digestive Diseases Week, American College of Gastroenterology, and United European Gastroenterology Week) and major thoracic meetings (American Thoracic Society Meeting, European Respiratory Society, and American Academy of Chest Physicians Conference) from 2013 to 2017. The review of abstracts and articles identified for full-text review was conducted via crowd sourcing using CrowdScreen SR 34 . Crowdsourcing has previously been shown to increase the efficiency of the systematic review process, while maintaining high accuracy during the review process 34 , 35 .…”

Section: Methodsmentioning

confidence: 99%

“…The review of abstracts and articles identified for full-text review was conducted via crowd sourcing using CrowdScreen SR 34 . Crowdsourcing has previously been shown to increase the efficiency of the systematic review process, while maintaining high accuracy during the review process 34 , 35 . Prior to screening, each member of the CrowdScreen SR Review team was asked to review a test set of 15 abstracts identified by the principal investigator (MEK).…”

Section: Methodsmentioning

confidence: 99%

Co-occurrence of Asthma and the Inflammatory Bowel Diseases: A Systematic Review and Meta-analysis

Kuenzig

Bishay

Leigh

et al. 2018

Clinical and Translational Gastroenterology

View full text Add to dashboard Cite

BackgroundInflammatory bowel diseases (IBD) and asthma share genetic and environmental risk factors. Consequently, several observational studies have explored an association between IBD and asthma. We systematically reviewed and summarized the literature on the co-occurrence of asthma and IBD.MethodsMEDLINE and EMBASE (to April 2017) were searched to identify observational studies on the association between asthma and IBD. Relative risks (RR) were pooled using random effects models. Heterogeneity was assessed using the I2 and Cochran Q statistics. Meta-regression based on study design, source of patients (population-based vs. tertiary-care center) and study location was conducted to explain between-study heterogeneity.ResultsEighteen studies were identified (15 Crohn’s disease, 15 ulcerative colitis (UC)). Asthma was associated with both Crohn’s disease (pooled RR 1.30, 95% confidence interval (CI) 1.16–1.47, I2 = 88%) and UC (RR 1.34, 95% CI 1.24–1.44, I2 = 93%). The study design and source of patients and study location explained between-study heterogeneity in Crohn’s disease, but not UC.ConclusionAsthma is associated with both Crohn’s disease and UC. Additional research is needed to determine if one disease influences the risk of developing the other or if the frequent co-occurrence of these diseases result from shared genetic, environmental, and microbial risk factors.

show abstract

“…Studies were excluded if they (1) were case reports or case series (fewer than ten patients), (2) performed vitamin D measurement after death (i.e., study on sudden infant death), (3) included adults but did not report study findings separately for children, and (4) were focused on specific diseases or interventions (i.e., cardiac surgery, prematurity or very low birth weight, acute lower respiratory tract infection, hematopoietic stem cell transplant). Interventional studies were not considered, because a recent scoping review of pediatric clinical trials (with an online searchable database) did not identify any potentially relevant publications [ 45 , 46 ].…”

Section: Methodsmentioning

confidence: 99%

“…Study eligibility was determined through two screening levels (Additional file 3 ). Title and abstract screening was performed by four authors (JDM, NN, KO, KI), followed by full-text review of potentially relevant citations by two independent authors (JDM, NN) using an online platform as previously described [ 45 ]. Disagreements between reviewers were resolved by consensus.…”

Section: Methodsmentioning

confidence: 99%

Vitamin D deficiency in critically ill children: a systematic review and meta-analysis

et al. 2017

Self Cite

View full text Add to dashboard Cite

BackgroundVitamin D deficiency (VDD) has been hypothesized not only to be common but also to represent a potentially modifiable risk factor for greater illness severity and clinical outcome during critical illness. The objective of this systematic review was to determine the frequency of VDD in pediatric critical illness and its association with clinical outcomes.MethodsMEDLINE, Embase, and CENTRAL were searched through December 12, 2016, with no date or language restrictions. The primary objective was to estimate the prevalence of VDD in the pediatric intensive care unit (PICU) and compare vitamin D status with healthy control populations. Secondary objectives were to evaluate whether VDD is associated with mortality, increased illness severity, PICU interventions, and patient clinical course. Random effects meta-analysis was used to calculate pooled VDD event rate, compare levels with those of control subjects, and evaluate for associations between VDD and clinical outcome.ResultsAmong 2700 citations, 17 studies meeting study eligibility were identified. The studies reported a total of 2783 critically ill children and had a median sample size of 120 (range 12–511). The majority of studies used a 25-hydroxyvitamin D [25(OH)D] level less than 50 nmol/L to define VDD, and the pooled VDD prevalence was 54.8 (95% CI 45.4–63.9). Average 25(OH)D levels were significantly lower in PICU patients than in healthy control subjects (pooled difference −17.3 nmol/L, 95% CI −14.0 to −20.6). In a meta-analysis calculation, we found that VDD was associated with increased mortality (OR 1.62, 95% CI 1.11–2.36), illness severity, and need for PICU interventions.ConclusionsApproximately 50% of critically ill children have VDD at the time of PICU admission, defined as a blood total 25(OH)D concentration under 50 nmol/L. VDD was further determined to be associated with greater illness severity, multiple organ dysfunction, and mortality in the PICU setting. Clinical trials are required to determine if optimization of vitamin D status improves patient outcome.Trial registrationPROSPERO, CRD42016026617. Registered on 11 January 2016.Electronic supplementary materialThe online version of this article (doi:10.1186/s13054-017-1875-y) contains supplementary material, which is available to authorized users.

show abstract

A pilot validation study of crowdsourcing systematic reviews: update of a searchable database of pediatric clinical trials of high-dose vitamin D

Cited by 22 publications

References 52 publications

Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study

Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study

Co-occurrence of Asthma and the Inflammatory Bowel Diseases: A Systematic Review and Meta-analysis

Vitamin D deficiency in critically ill children: a systematic review and meta-analysis

Contact Info

Product

Resources

About