The Reported Validity and Reliability of Methods for Evaluating Continuing Medical Education: A Systematic Review

Ratanawongsa, Neda; Thomas, Patricia A.; Marinopoulos, Spyridon S; Dorman, Todd; Wilson, Lisa M; Ashar, Bimal H.; Magaziner, Jeffrey; Miller, Redonda G.; Prokopowicz, Gregory; Qayyum, Rehan; Bass, Eric B.

doi:10.1097/acm.0b013e3181637925

Cited by 56 publications

(37 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On the other hand, teaching in clinical settings usually involves smaller groups of learners and a greater opportunity for group-based discussion, which, in turn, is enhanced by teachers who are approachable and by minimal psychological size differentials between teachers and learners. Systematic reviews of CME program evaluations have demonstrated the widespread use of methods with poor validity evidence, 26,27 a focus on learner outcomes, 28 and neglect of the higher Kirkpatrick outcomes of behavior and results. 27 Our study addresses these limitations by providing a measure of feedback for CME presenters that is based on observed behaviors and is supported by strong validity evidence.…”

Section: Discussionmentioning

confidence: 99%

Improving Participant Feedback to Continuing Medical Education Presenters in Internal Medicine: A Mixed-Methods Study

et al. 2011

View full text Add to dashboard Cite

BACKGROUND: Feedback is essential for improving the skills of continuing medical education (CME) presenters. However, there has been little research on improving the quality of feedback to CME presenters. OBJECTIVES: To validate an instrument for generating balanced and behavior-specific feedback from a national cross-section of participants to presenters at a large internal medicine CME course. DESIGN, SETTING, AND PARTICIPANTS: A prospective, randomized validation study with qualitative data analysis that included all 317 participants at a Mayo Clinic internal medicine CME course in 2009. MEASUREMENTS: An 8-item (5-point Likert scales) CME faculty assessment enhanced study form (ESF) was designed based on literature and expert review. Course participants were randomized to a standard form, a generic study form (GSF), or the ESF. The dimensionality of instrument scores was determined using factor analysis to account for clustered data. Internal consistency and interrater reliabilities were calculated. Associations between overall feedback scores and presenter and presentation variables were identified using generalized estimating equations to account for multiple observations within talk and speaker combinations. Two raters reached consensus on qualitative themes and independently analyzed narrative entries for evidence of balanced and behavior-specific comments. RESULTS: Factor analysis of 5,241 evaluations revealed a uni-dimensional model for measuring CME presenter feedback. Overall internal consistency (Cronbach alpha=0.94) and internal consistency reliability (ICC range 0.88-0.95) were excellent. Feedback scores were associated with presenters' academic ranks (mean score): Instructor (4.12), Assistant Professor (4.38), Associate Professor (4.56), Professor (4.70) (p=0.046). Qualitative analysis revealed that the ESF generated the highest numbers of balanced comments (GSF=11, ESF = 26; p = 0.01) and behavior-specific comments (GSF=64, ESF=104; p=0.001). CONCLUSIONS: We describe a practical and validated method for generating balanced and behavior-specific feedback for CME presenters in internal medicine. Our simple method for prompting course participants to give balanced and behavior-specific comments may ultimately provide CME presenters with feedback for improving their presentations.

show abstract

Section: Discussionmentioning

confidence: 99%

Improving Participant Feedback to Continuing Medical Education Presenters in Internal Medicine: A Mixed-Methods Study

et al. 2011

View full text Add to dashboard Cite

show abstract

“…16,17 Systematic reviews have documented that validity evidence is infrequently reported. Estimates vary widely depending on the sample, but content, internal structure, and relations with other variables evidence are typically reported in < 40 % of studies, 6,9,13,[18][19][20][21] and response process and consequences are reported in < 10 %. 18 However, the reporting of validity evidence for patient outcomes is unknown.…”

Section: Methodological Issues In Patient Outcomes Studiesmentioning

confidence: 99%

“…22,87,88 The reporting of validity evidence in this sample was even less frequent than in previous reviews in medical education. 6,9,13,[18][19][20][21] We discuss this below. Our findings regarding the prevalence of unit of analysis error mirror those in clinical medicine.…”

Section: Integration With Other Literaturementioning

confidence: 99%

Patient Outcomes in Simulation-Based Medical Education: A Systematic Review

et al. 2013

View full text Add to dashboard Cite

OBJECTIVES:Evaluating the patient impact of health professions education is a societal priority with many challenges. Researchers would benefit from a summary of topics studied and potential methodological problems. We sought to summarize key information on patient outcomes identified in a comprehensive systematic review of simulation-based instruction. DATA SOURCES: Systematic search of MEDLINE, EMBASE, CINAHL, PsychINFO, Scopus, key journals, and bibliographies of previous reviews through May 2011. STUDY ELIGIBILITY: Original research in any language measuring the direct effects on patients of simulationbased instruction for health professionals, in comparison with no intervention or other instruction. APPRAISAL AND SYNTHESIS: Two reviewers independently abstracted information on learners, topics, study quality including unit of analysis, and validity evidence. We pooled outcomes using random effects. RESULTS: From 10,903 articles screened, we identified 50 studies reporting patient outcomes for at least 3,221 trainees and 16,742 patients. Clinical topics included airway management (14 studies), gastrointestinal endoscopy (12), and central venous catheter insertion (8). There were 31 studies involving postgraduate physicians and seven studies each involving practicing physicians, nurses, and emergency medicine technicians. Fourteen studies (28 %) used an appropriate unit of analysis. Measurement validity was supported in seven studies reporting content evidence, three reporting internal structure, and three reporting relations with other variables. The pooled Hedges' g effect size for 33 comparisons with no intervention was 0.47 (95 % confidence interval [CI], 0.31-0.63); and for nine comparisons with non-simulation instruction, it was 0.36 (95 % CI, −0.06 to 0.78). LIMITATIONS: Focused field in education; high inconsistency (I 2 >50 % in most analyses). CONCLUSIONS: Simulation-based education was associated with small-moderate patient benefits in comparison with no intervention and non-simulation instruction, although the latter did not reach statistical significance. Unit of analysis errors were common, and validity evidence was infrequently reported.

show abstract

“…We used the same definitions for sources of reliability and validity evidence that were used in a prior medical education review. 8 Evidence sources for reliability included test-retest reliability such as reporting a correlation coefficient of scores from tests taken twice over a period of time by learners, a coefficient for internal consistency such as the Cronbach alpha, and inter-rater reliability such as intraclass correlation. Validity evidence required descriptions and intended purpose of the evaluation method and statistical or psychometric testing of learners from at least 1 evidence source among content, construct, or criterion validity.…”

Section: Methodsmentioning

confidence: 99%

Educational Testing Validity and Reliability in Pharmacy and Medical Education Literature

Hoover

Jung

Jacobs

et al. 2013

American Journal of Pharmaceutical Education

View full text Add to dashboard Cite

Objectives. To evaluate and compare the reliability and validity of educational testing reported in pharmacy education journals to medical education literature. Methods. Descriptions of validity evidence sources (content, construct, criterion, and reliability) were extracted from articles that reported educational testing of learners' knowledge, skills, and/or abilities. Using educational testing, the findings of 108 pharmacy education articles were compared to the findings of 198 medical education articles. Results. For pharmacy educational testing, 14 articles (13%) reported more than 1 validity evidence source while 83 articles (77%) reported 1 validity evidence source and 11 articles (10%) did not have evidence. Among validity evidence sources, content validity was reported most frequently. Compared with pharmacy education literature, more medical education articles reported both validity and reliability (59%; p,0.001). Conclusion. While there were more scholarship of teaching and learning (SoTL) articles in pharmacy education compared to medical education, validity, and reliability reporting were limited in the pharmacy education literature.Keywords: validity evidence, educational testing, pharmacy education, medical education INTRODUCTION Pharmacy educators use a wide variety of evaluation methods to ascertain whether students achieved specific learning objectives. When developing and evaluating the effectiveness of a doctor of pharmacy (PharmD) curriculum, educators must consider the standards for validity and reliability of educational testing.1 Standardized tests, such as the Pharmacy College Admission Test and North American Pharmacist Licensure Examination, are used as bookends to assess students' pharmacy-related knowledge and infer competence for licensure.2.3 Predictive evidence exists for these student performances.4-7 Educational testing throughout a PharmD program should provide valid and reliable assessment of students' abilities.When reporting evaluation methods used in the educational research of health professions, it is essential to consider evidence for validity and reliability. The authors were not aware of any literature reviews assessing the extent of validity and reliability reporting associated with evaluation methods in the pharmacy education literature. The objectives for this study were to characterize reliability and validity with educational testing reported in pharmacy education journals, and compare these with medical education literature reporting. METHODSWe evaluated validity and reliability reporting in articles that focused on educational testing of learner knowledge, skills, or abilities. To describe levels of reliability and validity reporting associated with pharmacy education literature, articles published in pharmacy education journals were reviewed and the findings were compared to medical education articles. Journals reviewed within pharmacy education were American Journal of Pharmaceutical Education (AJPE), Currents in Pharmacy Corresponding Author: Michael J. Peete...

show abstract

The Reported Validity and Reliability of Methods for Evaluating Continuing Medical Education: A Systematic Review

Cited by 56 publications

References 51 publications

Improving Participant Feedback to Continuing Medical Education Presenters in Internal Medicine: A Mixed-Methods Study

Improving Participant Feedback to Continuing Medical Education Presenters in Internal Medicine: A Mixed-Methods Study

Patient Outcomes in Simulation-Based Medical Education: A Systematic Review

Educational Testing Validity and Reliability in Pharmacy and Medical Education Literature

Contact Info

Product

Resources

About