Assessing Quality in Systematic Literature Reviews: A Study of Novice Rater Training

Abstract

<jats:p> This study investigated performance variability when graduate students critically appraised original studies from a systematic review. Fourteen doctoral students from different academic programs, with no systematic review experience, received training on the Methodological Quality Questionnaire (MQQ) rating scale. Participants were mostly male (71%) and non-native English speakers (79%). Each rater was randomly assigned one original study to independently assess using the MQQ. Their scores were compared to an expert rater. Statistical analysis comprised the following: percentage of agreement (POA), Kappa coefficient, and Kendall’s tau-b correlation. On the completed MQQ rating scale, 43% of the novice raters had a POA of 78% or higher with the expert rater. From this case study, a guide for improving training on methodological quality assessment was developed. Benefits include the following: (a) developing and supporting critical reasoning as well as problem-solving skills and (b) increasing research skills and competencies in the systematic review process. </jats:p>

Meta-analysis of the relationship between university students' anxiety and academic performance during the coronavirus disease 2019 pandemic

“…Previous researchers have used the Joanna Briggs Institute Methodological Quality Questionnaire (MQQ) framework (Acosta et al, 2020 ) to evaluate the methodological quality of the selected articles. This method quality checklist provides researchers with a basis for critical evaluation in terms of nine dimensions: theoretical definition, operational definition, research design, sample design, sample description, data credibility, data analysis, practical impact, and policy impact.…”

Challenges and critical success factors of digital communication, collaboration and knowledge sharing in project management virtual teams: a review

“…A quality assessment was also conducted on the extracted data. A series of quality assessment questions, based on the Methodological Quality Questionnaire [6], were answered to determine the extracted data's overall quality, The quality assessment questionnaire (see Appendix B) focused on assessing the source and author creditability, selection bias, methodological rigor, and usefulness in addressing the research question. Answers were graded as either "Yes", "No", "Can't Tell" or "Not Applicable", where factors to consider were listed under each criterion.…”

“…The importance of communication and knowledge sharing in any project-based environment is essential for project success, and this has become more evident with the rise of digital project execution [3]. Additionally, the drive toward project virtual teams (VT) has accelerated with the widespread proliferation of new digital communication technologies and the global COVID-19 pandemic [4][5][6]. A survey suggested that approximately 66% of multinational corporations utilize VTs, where 80% of the respondents believed this number would continue to increase [5].…”

Research Data Management Training in Academic Libraries: A Scoping Review

“…Some studies generate adequate contributions to the field, whereas others produce biases. Therefore, the quality of the research methodology of the included articles was explicitly and systematically assessed, specifically in the stage of full-text screening, using a scoring instrument adapted from the Methodological Quality Questionnaire (Acosta et al, 2020). This reflected the quality of the evidence within each study and provided a general score of methodological quality.…”