Setting Mastery Learning Standards

Abstract

Mastery learning is an instructional approach in which educational progress is based on demonstrated performance, not curricular time. Learners practice and retest repeatedly until they reach a designated mastery level; the final level of achievement is the same for all, although time to mastery may vary. Given the unique properties of mastery learning assessments, a thoughtful approach to establishing the performance levels and metrics that determine when a learner has demonstrated mastery is essential.Standard-setting procedures require modification when used for mastery learning settings in health care, particularly regarding the use of evidence-based performance data, the determination of appropriate benchmark or comparison groups, and consideration of patient safety consequences. Information about learner outcomes and past performance data of learners successful at the subsequent level of training can be more helpful than traditional information about test performance of past examinees. The marginally competent "borderline student" or "borderline group" referenced in traditional item-based and examinee-based procedures will generally need to be redefined in mastery settings. Patient safety considerations support conjunctive standards for key knowledge and skill subdomains and for items that have an impact on clinical outcomes. Finally, traditional psychometric indices used to evaluate the quality of standards do not necessarily reflect critical measurement properties of mastery assessments. Mastery learning and testing are essential to the achievement and assessment of entrustable professional activities and residency milestones. With careful attention, sound mastery standard-setting procedures can provide an essential step toward improving the effectiveness of health professions education, patient safety, and patient care.

Ensuring Competency of Novice Laparoscopic Surgeons—Exploring Standard Setting Methods and their Consequences

“…The average expert performance level is a method in which the pass/fail level is set at the median performance level of a group of experienced surgeons. 20 The contrasting groups method 17 sets the pass/fail level using the normal distribution of performance scores from 2 groups: competent and noncompetent. The pass/fail level is usually set at the intersection between the distributions of the 2 groups.…”

Effect of Trainee Performance Data on Standard-Setting Judgments Using the Mastery Angoff Method

“…Prior research has shown that giving judges baseline performance data from former trainees affects judgments during standard-setting exercises using traditional Angoff and Hofstee procedures. 11,13 The effect of giving judges prior examinee performance data during Mastery Angoff standard setting is not known.…”

“…12 However, these methods consider the borderline learner (Angoff) or a minimum and maximum failure rate (Hofstee), and are not appropriate for assessment when patient safety is a concern. Yudkowsky and colleagues 11 argued that, when determining an MPS in a mastery learning environment, judges should be asked to consider the performance of a trainee who is ''well prepared to succeed at the next stage of instruction or practice.'' Setting an acceptable failure rate or an expected pass rate is inappropriate, because all learners are expected to pass with sufficient, high-quality training.…”

“…9 Several standard-setting methods have been described, including the Angoff, Hofstee, Contrasting Groups, and Mastery Angoff. 10,11 The Angoff and Hofstee methods have traditionally been used for setting standards in most health care-related mastery learning curricula. 12 However, these methods consider the borderline learner (Angoff) or a minimum and maximum failure rate (Hofstee), and are not appropriate for assessment when patient safety is a concern.…”

“…11 In these situations, giving judges anchoring data may be irrelevant for several reasons. Anchoring data from traditionally trained participants' baseline tests do not necessarily inform performance prediction, and baseline tests typically have a low pass rate in mastery settings.…”