Quality assessment with diverse studies (QuADS): an appraisal tool for methodological and reporting quality in systematic reviews of mixed- or multi-method studies

Abstract

Background In the context of the volume of mixed- and multi-methods studies in health services research, the present study sought to develop an appraisal tool to determine the methodological and reporting quality of such studies when included in systematic reviews. Evaluative evidence regarding the design and use of our existing Quality Assessment Tool for Studies with Diverse Designs (QATSDD) was synthesised to enhance and refine it for application across health services research. Methods Secondary data were collected through a literature review of all articles identified using Google Scholar that had cited the QATSDD tool from its inception in 2012 to December 2019. First authors of all papers that had cited the QATSDD (n=197) were also invited to provide further evaluative data via a qualitative online survey. Evaluative findings from the survey and literature review were synthesised narratively and these data used to identify areas requiring refinement. The refined tool was subject to inter-rater reliability, face and content validity analyses. Results Key limitations of the QATSDD tool identified related to a lack of clarity regarding scope of use of the tool and in the ease of application of criteria beyond experimental psychological research. The Quality Appraisal for Diverse Studies (QuADS) tool emerged as a revised tool to address the limitations of the QATSDD. The QuADS tool demonstrated substantial inter-rater reliability (k=0.66), face and content validity for application in systematic reviews with mixed, or multi-methods health services research. Conclusion Our findings highlight the perceived value of appraisal tools to determine the methodological and reporting quality of studies in reviews that include heterogeneous studies. The QuADS tool demonstrates strong reliability and ease of use for application to multi or mixed-methods health services research.

Do patient engagement interventions work for all patients? A systematic review and realist synthesis of interventions to enhance patient safety

“…Due to the heterogeneity of the study types, the Quality Assessment Tool for Diverse Studies, a validated quality appraisal tool, 27 was used. Two reviewers (B. N. and K.…”

Positive Youth Development in the Context of Climate Change: A Systematic Review

“…Considering the included studies' heterogeneity, we used the Quality Assessment for Diverse Studies tool (QuADS, Harrison et al, 2021 ), a refined version of the Quality Assessment Tool for Studies with Diverse Designs (QATSDD, Sirriyeh et al, 2012 ). QuADS is used to determine the methodological and reporting quality and transparency of multi- and/or mixed-methods studies when included in systematic reviews.…”

Social Network Influences on Young Women's Choice to Use Long‐Acting Reversible Contraception: A Systematic Review

“…But the diversity of the study designs made comparison of results across studies challenging. The QATSDD was originally chosen because of its ability to assess the methodological quality of studies with mixed‐methods designs; however, after the completion of the current study, a new and revised tool known as the QuADS (Quality Assessment for Diverse Studies) emerged 69 . The QuADS intends to address limitations of the usability of the QATSDD in disciplines other than psychology.…”