Validity evidence for programmatic assessment in competency-based education

Abstract

IntroductionCompetency-based education (CBE) is now pervasive in health professions education. A foundational principle of CBE is to assess and identify the progression of competency development in students over time. It has been argued that a programmatic approach to assessment in CBE maximizes student learning. The aim of this study is to investigate if programmatic assessment, i. e., a system of assessment, can be used within a CBE framework to track progression of student learning within and across competencies over time.MethodsThree workplace-based assessment methods were used to measure the same seven competency domains. We performed a retrospective quantitative analysis of 327,974 assessment data points from 16,575 completed assessment forms from 962 students over 124 weeks using both descriptive (visualization) and modelling (inferential) analyses. This included multilevel random coefficient modelling and generalizability theory.ResultsRandom coefficient modelling indicated that variance due to differences in inter-student performance was highest (40%). The reliability coefficients of scores from assessment methods ranged from 0.86 to 0.90. Method and competency variance components were in the small-to-moderate range.DiscussionThe current validation evidence provides cause for optimism regarding the explicit development and implementation of a program of assessment within CBE. The majority of the variance in scores appears to be student-related and reliable, supporting the psychometric properties as well as both formative and summative score applications.

Competency assessment of an undergraduate program using a third-party, objective pre-post examination

“…Second, when assessing competencies, leaders need to evaluate program metrics over time. These repeated measures help assess reliability of the program and control for the largest variance producer in at least one study, the student [ 39 ]. Without such a baseline and repeated measurements, it is impossible to gauge the effectiveness of teaching on competency attainment.…”

Development of core entrustable professional activities linked to a competency-based veterinary education framework

“…Identifying the competencies required to perform these EPAs creates links to the competency framework, demonstrates application of competencies, and provides useful information for administrative purposes (e.g., curricular mapping, accreditation). Furthermore, linking the EPAs to an entrustment scale and other workplace assessments enables targeted feedback on learner progression in the "real life" teaching and workplace context of veterinary clinicians (Ten Cate et al 2016;Van Melle et al 2017;Bok et al 2018). Correspondingly, this article reports the process used by an international team of veterinary educators to create core clinical EPAs mapped to a recently-created CBVE Framework.…”

Outcomes-Based Selection Into Medical School: Predicting Excellence in Multiple Competencies During the Clinical Years

“… 12 Importantly, many finer-grained indicators are available for measuring performance in the clinical years of medical school, including performance on formal and workplace-based assessments using competencies or EPAs. 13 – 18 These finer-grained metrics may be more authentic and more informative indicators of success during the clinical years of medical school. 19 …”