A systematic review of the content of critical appraisal tools

Abstract

BackgroundConsumers of research (researchers, administrators, educators and clinicians) frequently use standard critical appraisal tools to evaluate the quality of published research reports. However, there is no consensus regarding the most appropriate critical appraisal tool for allied health research. We summarized the content, intent, construction and psychometric properties of published, currently available critical appraisal tools to identify common elements and their relevance to allied health research.MethodsA systematic review was undertaken of 121 published critical appraisal tools sourced from 108 papers located on electronic databases and the Internet. The tools were classified according to the study design for which they were intended. Their items were then classified into one of 12 criteria based on their intent. Commonly occurring items were identified. The empirical basis for construction of the tool, the method by which overall quality of the study was established, the psychometric properties of the critical appraisal tools and whether guidelines were provided for their use were also recorded.ResultsEighty-seven percent of critical appraisal tools were specific to a research design, with most tools having been developed for experimental studies. There was considerable variability in items contained in the critical appraisal tools. Twelve percent of available tools were developed using specified empirical research. Forty-nine percent of the critical appraisal tools summarized the quality appraisal into a numeric summary score. Few critical appraisal tools had documented evidence of validity of their items, or reliability of use. Guidelines regarding administration of the tools were provided in 43% of cases.ConclusionsThere was considerable variability in intent, components, construction and psychometric properties of published critical appraisal tools for research reports. There is no "gold standard' critical appraisal tool for any study design, nor is there any widely accepted generic tool that can be applied equally well across study types. No tool was specific to allied health research requirements. Thus interpretation of critical appraisal of research reports currently needs to be considered in light of the properties and intent of the critical appraisal tool chosen for the task.

Risk factors for healthcare-associated infection in pediatric intensive care units: a systematic review

“…Although the importance of critically appraising articles is well recognized, there is no consensus on valid indicators for the quality of observational studies 113 . In the analysis of quality proposed by the Newcastle-Ottawa Scale, studies can obtain the maximum score even when presenting a number of sources of heterogeneity and bias.…”

HIV Prevention Interventions to Reduce Racial Disparities in the United States: A Systematic Review

“…Comparing our review to other reviews is limited by the use of a unique modification of the DB scoring system used in this symposium which was chosen to allow the evaluation of both randomized and non-randomized studies. 53 However, the original DB system has been used extensively by reviews in other health settings, with the original paper having been cited over 800 times since publication. It is worth emphasizing that the DB scoring metric evaluates the quality of the study evaluating an intervention, not the intervention itself.…”

The efficacy of community-based rehabilitation programmes for adults with TBI

“…The McMaster tool was selected as it relates specifi cally to allied health research; it can be used to appraise both qualitative and quantitative research; it includes clear guidelines on the interpretation of each aspect of critical appraisal, and the appraisal process results in a non-numerical summary of overall study quality. All critical appraisal tools have limitations (Katrak, 2004); it was therefore decided to create an additional quality checklist based on the synthesis of four critical appraisal tools (Law et al, 1998;Batavia, 2001;Walsh and Wigens, 2003;Humphris, 2005), which addressed aspects of research quality not included in the McMaster tool. The overall quality of each study was determined using a combination of the detailed appraisal from the McMaster form, and the quality checklist.…”