How credible are the study results? Evaluating and applying internal validity tools to literature-based assessments of environmental health hazards

Abstract

Environmental health hazard assessments are routinely relied upon for public health decision-making. The evidence base used in these assessments is typically developed from a collection of diverse sources of information of varying quality. It is critical that literature-based evaluations consider the credibility of individual studies used to reach conclusions through consistent, transparent and accepted methods. Systematic review procedures address study credibility by assessing internal validity or “risk of bias” — the assessment of whether the design and conduct of a study compromised the credibility of the link between exposure/intervention and outcome. This paper describes the commonalities and differences in risk-of-bias methods developed or used by five groups that conduct or provide methodological input for performing environmental health hazard assessments: the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group, the Navigation Guide, the National Toxicology Program’s (NTP) Office of Health Assessment and Translation (OHAT) and Office of the Report on Carcinogens (ORoC), and the Integrated Risk Information System of the U.S. Environmental Protection Agency (EPA-IRIS). Each of these groups have been developing and applying rigorous assessment methods for integrating across a heterogeneous collection of human and animal studies to inform conclusions on potential environmental health hazards. There is substantial consistency across the groups in the consideration of risk-of-bias issues or “domains” for assessing observational human studies. There is a similar overlap in terms of domains addressed for animal studies; however, the groups differ in the relative emphasis placed on different aspects of risk of bias. Future directions for the continued harmonization and improvement of these methods are also discussed.

Making nutrition guidelines fit for purpose

“…GRADE is being modified for application to environmental health topics 37. For example, some frameworks for assessing research on environmental hazards start with an initial higher rating for observational studies than GRADE would apply 38. Importantly, the process for formulating the research questions is based on criteria such as priority and uncertainty rather than the availability of certain types of evidence or the methods needed to synthesise it 34…”

NTP Monograph on the Systematic Review of Traffic-related Air Pollution and Hypertensive Disorders of Pregnancy

“…In the OHAT approach, some risk-of-bias questions or elements are considered potentially more important when assessing studies because there is more empirical evidence that these areas of bias have a greater effect on estimates of the effect size or because these issues are generally considered to have a greater effect on the credibility of study results in environmental health studies (Rooney et al 2016). There were three key questions for observational human studies: confounding, exposure characterization, and outcome assessment.…”

Reproducibility and Research Integrity

“…Reproducibility provides scientists with evidence that research results are objective and reliable and not due to bias or chance (Rooney et al 2016). Irreproducibility, by contrast, may indicate a problem with any of the steps involved in the research such as, but not limited to, the experimental design, variability of biological materials (such as cells, tissues or animal or human subjects), data quality or integrity, statistical analysis, or study description (Landis et al 2012, Shamoo and Resnik 2015).…”

“…Scientific journals, funding agencies, and researchers have responded to the reproducibility “crisis” by articulating standards for designing experiments, analyzing data, and reporting methods, materials, data, and results (Landis et al 2012, Pusztai et al 2013, Collins and Tabak 2014, McNutt 2014, The Science Exchange Network 2014, Nature 2014a, National Institutes of Health 2016, Rooney et al 2016). While many of these standards tend to be discipline-specific, some apply across disciplines.…”

“…controlled trial, prospective cohort study), methods (e.g. blinding, randomization), procedures, techniques, materials, equipment, data analysis methods and tools (including computer programs or codes), study population (for animals or humans), exclusion and inclusion criteria (for animals and humans), ethics committee approvals (if appropriate), theoretical assumptions and potential biases, sources of funding, and conflicts of interest (Landis et al 2012, Nature 2014b, McNutt 2014, Nature 2015, Elliott and Resnik 2015, Rooney et al 2016, Morgan et al 2016). Additional disclosures may need to occur after the research is published to allow independent scientists to obtain information needed to reproduce experiments, reanalyze data, or develop new hypotheses or theories related to the research.…”