Best Evidence in Emergency Medicine (<scp>BEEM</scp>) Rater Scores Correlate With Publications’ Future Citations

Carpenter, Christopher R.; Sarli, Cathy; Fowler, Susan A.; Kulasegaram, Kulamakan; Vallera, Teresa; Lapaine, Pierre; Schalet, Grant; Worster, Andrew

doi:10.1111/acem.12235

Cited by 30 publications

(25 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Much akin to other crowd-based rating systems (e.g. BEEM rating score27–28 and Yelp), group-based decision-making ultimately may be the best guide for rating individual resources.…”

Section: Discussionmentioning

confidence: 99%

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study

Chan¹,

Thoma²,

Krishnan³

et al. 2016

WestJEM

Self Cite

View full text Add to dashboard Cite

IntroductionOnline education resources (OERs), like blogs and podcasts, increasingly augment or replace traditional medical education resources such as textbooks and lectures. Trainees’ ability to evaluate these resources is poor, and few quality assessment aids have been developed to assist them. This study aimed to derive a quality evaluation instrument for this purpose.MethodsWe used a three-phase methodology. In Phase 1, a previously derived list of 151 OER quality indicators was reduced to 13 items using data from published consensus-building studies (of medical educators, expert podcasters, and expert bloggers) and subsequent evaluation by our team. In Phase 2, these 13 items were converted to seven-point Likert scales used by trainee raters (n=40) to evaluate 39 OERs. The reliability and usability of these 13 rating items was determined using responses from trainee raters, and top items were used to create two OER quality evaluation instruments. In Phase 3, these instruments were compared to an external certification process (the ALiEM AIR certification) and the gestalt evaluation of the same 39 blog posts by 20 faculty educators.ResultsTwo quality-evaluation instruments were derived with fair inter-rater reliability: the METRIQ-8 Score (Inter class correlation coefficient [ICC]=0.30, p<0.001) and the METRIQ-5 Score (ICC=0.22, p<0.001). Both scores, when calculated using the derivation data, correlated with educator gestalt (Pearson’s r=0.35, p=0.03 and r=0.41, p<0.01, respectively) and were related to increased odds of receiving an ALiEM AIR certification (odds ratio=1.28, p=0.03; OR=1.5, p=0.004, respectively).ConclusionTwo novel scoring instruments with adequate psychometric properties were derived to assist trainees in evaluating OER quality and correlated favourably with gestalt ratings of online educational resources by faculty educators. Further testing is needed to ensure these instruments are accurate when applied by trainees.

show abstract

“…Much akin to other crowd-based rating systems (e.g. BEEM rating score27–28 and Yelp), group-based decision-making ultimately may be the best guide for rating individual resources.…”

Section: Discussionmentioning

confidence: 99%

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study

Chan¹,

Thoma²,

Krishnan³

et al. 2016

WestJEM

Self Cite

View full text Add to dashboard Cite

show abstract

“…The aim of this study was to determine which of the quality indicators previously documented in the literature were the most relevant. Given the track record of other scores 24 and processes, 25 we believe that a functional assessment instrument should be developed. We plan to use the data from this study and a modified Delphi of expert medical educators to identify vague or redundant quality indicators and create an assessment instrument that focuses on the essential components of quality.…”

Section: Discussionmentioning

confidence: 99%

Emergency Medicine and Critical Care Blogs and Podcasts: Establishing an International Consensus on Quality

Thoma

Chan

Paterson

et al. 2015

Annals of Emergency Medicine

View full text Add to dashboard Cite

Twitter: @Brent_Thoma. Study objective: This study identified the most important quality indicators for online educational resources such as blogs and podcasts.Methods: A modified Delphi process that included 2 iterative surveys was used to build expert consensus on a previously defined list of 151 quality indicators divided into 3 themes: credibility, content, and design. Aggregate social media indicators were used to identify an expert population of editors from a defined list of emergency medicine and critical care blogs and podcasts. Survey 1 consisted of the quality indicators and a 7-point Likert scale. The mean score for each quality indicator was included in survey 2, which asked participants whether to "include" or "not include" each quality indicator. The cut point for consensus was defined at greater than 70% "include."Results: Eighty-three percent (20/24) of bloggers and 90.9% (20/22) of podcasters completed survey 1 and 90% (18/20) of bloggers and podcasters completed survey 2. The 70% inclusion criteria were met by 44 and 80 quality indicators for bloggers and podcasters, respectively. Post hoc, a 90% cutoff was used to identify a list of 14 and 26 quality indicators for bloggers and podcasters, respectively. Conclusion:The relative importance of quality indicators for emergency medicine blogs and podcasts was determined. This will be helpful for resource producers trying to improve their blogs or podcasts and for learners, educators, and academic leaders assessing their quality. These results will inform broader validation studies and attempts to develop user-friendly assessment instruments for these resources.

show abstract

“…The scoring instrument contains five measurement outcomes using seven-point Likert scales: Best Evidence in Emergency Medicine (BEEM) score, accuracy, educational utility, evidence based, and references (Table 1). 8 More detailed methods are described in the original description of the AIR series 7. Board members with any role in the production of a reviewed resource recused him/herself from grading that resource.…”

Section: Methodsmentioning

confidence: 99%

Blog and Podcast Watch: Neurologic Emergencies

Grock

Joshi

Swaminathan

et al. 2016

WestJEM

View full text Add to dashboard Cite

IntroductionThe WestJEM Blog and Podcast Watch presents high quality open-access educational blogs and podcasts in emergency medicine (EM) based on the ongoing ALiEM Approved Instructional Resources (AIR) and AIR-Professional series. Both series critically appraise resources using an objective scoring rubric. This installment of the Blog and Podcast Watch highlights the topic of neurologic emergencies from the AIR series.MethodsThe AIR series is a continuously building curriculum that follows the Council of Emergency Medicine Residency Director’s (CORD) annual testing schedule. For each module, relevant content is collected from the top 50 Social Media Index sites published within the previous 12 months, and scored by eight board members using five equally weighted measurement outcomes: Best Evidence in Emergency Medicine (BEEM) score, accuracy, educational utility, evidence based, and references. Resources scoring ≥30 out of 35 available points receive an AIR label. Resources scoring 27–29 receive an honorable mention label, if the executive board agrees that the post is accurate and educationally valuable.ResultsA total of 125 blog posts and podcasts were evaluated. Key educational pearls from the 14 AIR posts are summarized, and the 20 honorable mentions are listed.ConclusionThe WestJEM Blog and Podcast Watch series is based on the AIR and AIR-Pro series, which attempts to identify high quality educational content on open-access blogs and podcasts. This series provides an expert-based, post-publication curation of educational social media content for EM clinicians with this installment focusing on neurologic emergencies.

show abstract

Best Evidence in Emergency Medicine (BEEM) Rater Scores Correlate With Publications’ Future Citations

Cited by 30 publications

References 54 publications

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study

Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study

Emergency Medicine and Critical Care Blogs and Podcasts: Establishing an International Consensus on Quality

Blog and Podcast Watch: Neurologic Emergencies

Contact Info

Product

Resources

About