Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis

Page, Matthew J.; Nguyen, Phi‐Yen; Hamilton, Daniel G; Haddaway, Neal R.; Kanukula, Raju; Moher, David; McKenzie, Joanne E.

doi:10.1016/j.jclinepi.2022.03.003

Cited by 30 publications

(34 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…We downloaded all identified OA journal articles in full text from the EPMC and processed them with the metareadr R package ( Serghiou et al 2021 ). While we assumed that OA articles available in the EPMC were more likely to contain open data ( Page et al 2022 ), we also checked non-OA journals. Thus, we randomly selected 500 non-OA articles.…”

Section: Methodsmentioning

confidence: 99%

Dental Research Data Availability and Quality According to the FAIR Principles

et al. 2022

View full text Add to dashboard Cite

According to the FAIR principles, data produced by scientific research should be findable, accessible, interoperable, and reusable—for instance, to be used in machine learning algorithms. However, to date, there is no estimate of the quantity or quality of dental research data evaluated via the FAIR principles. We aimed to determine the availability of open data in dental research and to assess compliance with the FAIR principles (or FAIRness) of shared dental research data. We downloaded all available articles published in PubMed-indexed dental journals from 2016 to 2021 as open access from Europe PubMed Central. In addition, we took a random sample of 500 dental articles that were not open access through Europe PubMed Central. We assessed data sharing in the articles and compliance of shared data to the FAIR principles programmatically. Results showed that of 7,509 investigated articles, 112 (1.5%) shared data. The average (SD) level of compliance with the FAIR metrics was 32.6% (31.9%). The average for each metric was as follows: findability, 3.4 (2.7) of 7; accessibility, 1.0 (1.0) of 3; interoperability, 1.1 (1.2) of 4; and reusability, 2.4 (2.6) of 10. No considerable changes in data sharing or quality of shared data occurred over the years. Our findings indicated that dental researchers rarely shared data, and when they did share, the FAIR quality was suboptimal. Machine learning algorithms could understand 1% of available dental research data. These undermine the reproducibility of dental research and hinder gaining the knowledge that can be gleaned from machine learning algorithms and applications.

show abstract

Section: Methodsmentioning

confidence: 99%

Dental Research Data Availability and Quality According to the FAIR Principles

et al. 2022

View full text Add to dashboard Cite

show abstract

“…While all data for a meta-analysis is typically summarised in a tables or forest plots, sharing an editable file containing extracted data (e.g., CSV, RevMan) reduces the time and risk of errors associated with manual extraction of such data. This then facilitates the review’s reuse in future updates and replications, or its inclusion in overviews of reviews and clinical practice guidelines [17,18]. Infrequent sharing of data in systematic reviews in health research has been observed, but these findings may not generalise to all health topics [4] or across journals [7].…”

Section: Introductionmentioning

confidence: 99%

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Nguyen

Kanukula

McKenzie

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Objectives: To examine changes in completeness of reporting and frequency of sharing review materials in systematic reviews (SRs) over time; and factors associated with these changes. Methods: We examined a random sample of 300 SRs with meta-analysis indexed in PubMed, Science Citation Index, Social Sciences Citation Index, Scopus and Education Collection in November 2020. We compared the extent of complete reporting in these reviews against 110 SRs indexed in MEDLINE in February 2014. We examined associations between completeness of reporting and various factors (e.g. self-reported use of reporting guidelines, journal’s data sharing policies) by calculating risk ratios (RR) and 95% confidence intervals (CI). Results: Among 300 SRs from 2020, authors of only 7% shared their review data file(s) and 1% shared analytic code. Compared to 2014 sample, reviews in 2020 were more likely to reference reporting guidelines in their manuscript (RR=2.8, 95% CI 2.1-3.8), report the full search strategy for at least one database (RR=1.3, 95% CI 1.1-1.6) and methods of data preparation for meta-analysis (RR=2.2, 95% CI 1.4-3.5). Among reviews in 2020, those for which authors mentioned using reporting guidelines reported review protocols, study screening processes and numbers of records retrieved for each database more frequently than those that did not mention reporting guidelines; however the 95% CIs for these associations included the null. Reviews published in journals that mandated either data sharing or inclusion of Data Availability Statements were more likely to share their review materials (e.g. data, code files) (RR=8.1, 95% CI 3.1-21.5). Conclusion: There was a notable increase over time in self-reported use of a reporting guideline, but we were uncertain whether it was associated with improved reporting of SRs. Data sharing policies of journals potentially encourage sharing of review materials. Further studies are needed to explore other facilitators or barriers to complete reporting in SRs.

show abstract

“…Recent studies have shown that author-reporting on the availability and accessibility of restricted data in publications is poor, particularly with respect to providing clear language and information in journal data availability statements (Gabelica et al, 2022;Page et al, 2022;Read et al, 2021). A previous study completed by the corresponding author found that when CIHRfunded authors reported that data could be shared or an application was required, none provided details about how to access the data (Read et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

Understanding the Challenges Associated with Finding and Accessing Restricted Data in Canada: A Mixed Methods Study

Read

Gibson²,

Leahey³

et al. 2022

Preprint

View full text Add to dashboard Cite

INTRODUCTIONThis study aimed to identify Canadian access-limited data sources and evaluate a subset of restricted health sciences data sources to determine how well they make their data discoverable and accessible.MATERIALS AND METHODSA search was conducted across Canadian sectors and national experts were consulted to identify access-limited data sources. A subset of restricted health sciences data sources (n=48) was evaluated using a rubric to assess how well they make their data discoverable and accessible. The rubric assigned data sources a grade of A through C denoting how well they met certain discoverability and access criteria. The degree to which data sources demonstrated consistency between their discovery and access grades was assessed using Kendall’s rank correlation coefficient.RESULTS137 Canadian data sources were identified. Restricted health sciences sources received poor data discovery grades due to a lack of metadata (38/48, 79%), an inability to search/browse datasets (32/46, 70%), and lack of data documentation to support interpretability and reuse (27/48, 56%). Low data accessibility grades were assigned for the lack of transparent pricing information (31/48, 65%) and opaque data restriction criteria (25/48, 52%). Data sources with higher discovery scores had higher access scores on average (tau-b=0.31, p=0.0059).DISCUSSIONThis study highlights areas for improvement with respect to the discovery of and access to Canadian restricted data. Developing metadata standards to accommodate restricted data access procedures, improving data infrastructure to support restricted data, and expanding data documentation training for restricted data custodians are necessary to ensure that restricted data is discoverable, accessible, and reusable in the future.

show abstract

Data and code availability statements in systematic reviews of interventions were often missing or inaccurate: a content analysis

Cited by 30 publications

References 30 publications

Dental Research Data Availability and Quality According to the FAIR Principles

Dental Research Data Availability and Quality According to the FAIR Principles

Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study

Understanding the Challenges Associated with Finding and Accessing Restricted Data in Canada: A Mixed Methods Study

Contact Info

Product

Resources

About