Learning Analytics in Medical Education Assessment: The Past, the Present, and the Future

Abstract

With the implementation of competency-based medical education (CBME) in emergency medicine, residency programs will amass substantial amounts of qualitative and quantitative data about trainees' performances. This increased volume of data will challenge traditional processes for assessing trainees and remediating training deficiencies. At the intersection of trainee performance data and statistical modeling lies the field of medical learning analytics. At a local training program level, learning analytics has the potential to assist program directors and competency committees with interpreting assessment data to inform decision making. On a broader level, learning analytics can be used to explore system questions and identify problems that may impact our educational programs. Scholars outside of health professions education have been exploring the use of learning analytics for years and their theories and applications have the potential to inform our implementation of CBME. The purpose of this review is to characterize the methodologies of learning analytics and explore their potential to guide new forms of assessment within medical education.

What we measure … and what we should measure in medical education

“…As much of the validity argument rests with providing evidence that performance at a certain point in time is related to future performance, the development of longitudinal databases, in which assessment results can be linked to an individual, is essential. Fortunately, many medical schools and residency programmes have developed information systems that allow for the storage and retrieval of longitudinal data concerning progress through the curriculum and the application of learning analytics . Likewise, with electronic medical records, unique provider identifiers and access to patient records, it has become easier to conduct studies that specifically link assessment data and quality of care …”

“…Fortunately, many medical schools and residency programmes have developed information systems that allow for the storage and retrieval of longitudinal data concerning progress through the curriculum and the application of learning analytics. 49 Likewise, with electronic medical records, unique provider identifiers and access to patient records, it has become easier to conduct studies that specifically link assessment data and quality of care. 50…”

Developing a dashboard to meet Competence Committee needs: a design-based research project

“…BT reviewed the literature on CC function, 11,16,[31][32][33] learning analytics, 12,21 and data visualization. 18,22,23,27,34 He summarized this work for our team's programming experts (DJ and VK) to provide the context required to support dashboard programming.…”

“…10,11 However, the transition to CBME is resulting in volumes of assessment data that dwarf those seen in traditional assessment programs. 12 CCs are struggling to make robust, data-driven decisions while also providing feedback that fosters resident development. [11][12][13][14][15][16] To realize the promise of CBME, resident assessment data need to be organized and displayed effectively 17,18 Analytical and visualization techniques have been developed in other fields (e.g.…”

“…sport and business) to address the challenges presented by large datasets. 12 The subfield of learning analytics uses large datasets, statistical techniques, and predictive modeling 19 to describe, characterize, and predict the learning behaviour of individuals 20,21 Dynamic dashboards, described as "a visual display of the most important information needed to achieve one or more objectives" are frequently used to consolidate and arrange these data so the information can be monitored at a glance". 17,22 The development of such dashboards is an iterative process 18 and requires collaboration with information technology experts, assessment experts, data managers, and dashboard users.…”