The Revised METRIQ Score: A Quality Evaluation Tool for Online Educational Resources

Abstract

Background: With the rapid proliferation of online medical education resources, quality evaluation is increasingly critical. The Medical Education Translational Resources: Impact and Quality (METRIQ) study evaluated the METRIQ-8 quality assessment instrument for blogs and collected feedback to improve it.Methods: As part of the larger METRIQ study, participants rated the quality of five blog posts on clinical emergency medicine topics using the eight-item METRIQ-8 score. Next, participants used a 7-point Likert scale and free-text comments to evaluate the METRIQ-8 score on ease of use, clarity of items, and likelihood of recommending it to others. Descriptive statistics were calculated and comments were thematically analyzed to guide the development of a revised METRIQ (rMETRIQ) score.Results: A total of 309 emergency medicine attendings, residents, and medical students completed the survey.The majority of participants felt the METRIQ-8 score was easy to use (mean AE SD = 2.7 AE 1.1 out of 7, with 1 indicating strong agreement) and would recommend it to others (2.7 AE 1.3 out of 7, with 1 indicating strong agreement). The thematic analysis suggested clarifying ambiguous questions, shortening the 7-point scale, specifying scoring anchors for the questions, eliminating the "unsure" option, and grouping-related questions. This analysis guided changes that resulted in the rMETRIQ score.Conclusion: Feedback on the METRIQ-8 score contributed to the development of the rMETRIQ score, which has improved clarity and usability. Further validity evidence on the rMETRIQ score is required.W ith increasing expansion of emergency medicine (EM) blogs and podcasts, residents frequently use these open educational resources to supplement and potentially replace traditional tools. 1-4 Unlike textbooks and journals, these online resources are rarely peer-reviewed 5-7 and critics raise concerns that learners From the

Evaluating the reliability of gestalt quality ratings of medical education podcasts: A METRIQ study

“…However, the ratings of small numbers of raters are insufficiently reliable. Further, our findings emphasize the need to develop tools that support podcast quality evaluation, which could build on the advances in quality evaluation of other open educational resources [ 12 – 14 , 24 ].…”

“…Studies have found the assessment of the quality of online resources to be difficult [7][8][9][10][11]. Resources have been developed to assist trainees and clinicians to assess the quality of blog posts [7,[12][13][14] but podcast listeners have had to rely upon their own gestalt to evaluate the quality of these resources. As the reliability of gestalt is limited by each individual's unique experience and learning needs [15,16], the effectiveness of this approach is unclear.…”

“…We hypothesized that, like the gestalt evaluation of blog post quality [8,12], clinicians will have broadly discrepant perspectives on the quality of individual podcasts. To test this hypothesis, we recruited an international, multidisciplinary sample of emergency clinicians to rate the quality of podcasts.…”

Consensus Guidelines for Digital Scholarship in Academic Promotion

“…Some scholars in this space have proposed that we move beyond bibliometrics and surrogates for quality (eg, impact factor, citations, altmetrics), and that P&T committees consider applying direct quality assessments to items of interest (eg, applying the revised METRIQ 41 or ALiEM Approved Instructional Resources (AIR series) scores 39 to a few choice works of digital scholarship from a faculty member’s CV, or applying the PRISMA 42 reporting guidelines to a few systematic reviews). Equitably applying both descriptive bibliometrics (eg, citation rate, h-index, etc.)…”

Role of Social Media in Health Professions Education

“…In addition to relying on users' critical appraisal skills, there are suggested approaches and quality indicators (Lin et al 2015;Thoma et al 2018b;Colmers-Gray et al 2019). One systematic review identified three core themes for qualitycredibility (transparency, process, use of other resources, trustworthiness, and bias), content (professionalism, engagement, academic rigor, and orientation), and design (aesthetics, interaction, functionality, and ease of use) (Paterson et al 2015).…”