Nuance and Noise: Lessons Learned From Longitudinal Aggregated Assessment Data

Abstract

Our model suggests that residents begin at different points and progress at different rates. Meta-raters such as program directors and Clinical Competency Committee members should bear in mind that progression may take time and learning trajectories will be nuanced. Individuals involved in ratings should be aware of sources of noise in the system, including the raters themselves.

Learning Analytics in Medical Education Assessment: The Past, the Present, and the Future

“…However, the systematic application of learning analytics to interpret these data is sporadic. Some groups have started using learning analytics to gather information and gain insights about learner‐ or system‐level performance (e.g., the McMaster Modular Assessment Program for emergency medicine, online modules that using learning analytics methods to teach x‐ray interpration, and an internal medicine program's analytics dashboards). The vast majority of residency programs, however, are attempting to execute programmatic assessment (i.e., the integrated system of multiple, longitudinal observations from multiple observers, aggregated into summary performance scores for group adjudication of global judgment) without optimized data collection (e.g., valid testing/simulations, timely and accurate workplace‐based assessments), modern analytic techniques, or appropriate data representation.…”

“…Similarly, when we make decisions about a trainee based on our data analyses, we must also bear in mind the consequences of our decisions affecting their later behaviors and performances, both good and bad. While early identification of trainees at risk may be valuable so programs can offer earlier personalized remediation plans, the very act of labeling a “trainee‐at‐risk” may have consequences toward their self‐perceptions and future performance . At the same time, the implications of not collecting and interpreting data may also be problematic.…”

“…Without timely data inputs (e.g., continuous workplace‐based assessment data from every day/shift in a rotation), we may be glossing over problems via acts of omission . It is critical, then, that those who are charged with aggregating and interpreting larger swaths of data to be aware of potential pitfalls of missing data or cognitive biases (e.g., halo effect) that may affect the meta‐rating (i.e., interpretation of multiple ratings from multiple raters) of the presented data.…”

“…Larger‐scale data sharing may also allow us to complete analyses that can inform broader policy decisions (e.g., human resources planning, funding) . For instance, if we can predict how long it takes for a typical trainee to progress through a system compared to how a gifted or struggling trainee might progress, we can begin to anticipate what sort of additional funding or new training opportunities are needed to ensure adequate training for all.…”

Education scholarship in Canadian emergency medicine: The past, present, and future

“…With this transition, scholarship is essential to inform the rollout refinement of a completely different curriculum. The new emphasis on multiple, low stakes, workplace-based assessments is supported by emerging [12][13][14] literature that asks more questions than it provides answers.…”