The Radar Graph: the Development of an Educational Tool to Demonstrate Resident Competency

Abstract

Background Despite the movement toward competency-based assessment by accrediting bodies in recent years, there is no consensus on how to best assess medical competence. Direct observation is a useful tool. At the same time, a comprehensive assessment system based on direct observation has been difficult to develop. Intervention We developed a system that translates data obtained from checklists of observed behaviors completed during educational activities, including direct observation of clinical care, into a graphic tool (the “radar graph”) usable for both formative and summative assessment. Using unique, observable behaviors to evaluate levels of competency on the Dreyfus scale, we assessed resident performance in 6 learning sites within our residency. Data are represented on a radar graph, which residents and faculty used to recognize both strengths and areas for growth to guide educational planning for the individual learner. Results Initial data show that the radar graphs have construct validity because the development process accurately reflects the desired construct, assessors were adequately trained, and the radar graphs demonstrated resident growth over time. A form completion rate of 90% for >1500 disseminated assessments suggests the feasibility of our process. Conclusions The radar graph is a promising tool for use in resident feedback and competency assessment. Further research is needed to determine the full utility of the radar graphs, including a better understanding of the tool's reliability and construct validity.

Psychometric validation of the Laval Developmental Benchmarks Scale for Family Medicine

“…The empirical data show that year 1 residents have better CAS than expected and that the transition from year 1 to year 2 is much more gradual. This gradual increase in competency level throughout residency training was also observed in another study (33). A possible explanation for these divergent results in the literature is that the progression of residents' competency levels could be highly dependent on the program.…”

“…The large distance in logits between the two steps of the rating scale suggests that there is space on the latent competency variable for a ner-grained rating scale. For instance, a rating scale similar to the O-SCORE could be considered (33). The O-SCORE has 5 levels that re ect a ner grained progression toward complete independence.…”

Psychometric validation of the Laval developmental benchmarks scale for family medicine

“…In addition, the highest rating (option 3) on the rating scale could be split into two or three options, with the highest option going beyond “Independent.” The large distance in logits between the two steps of the rating scale suggests that there is space on the latent competency variable for a finer-grained rating scale. For instance, a rating scale similar to the O-SCORE could be considered [ 32 ]. The O-SCORE has 5 levels that reflect a finer grained progression toward complete independence.…”

Engaging tertiary educators in the development of their assessment literacy

“…Rubrics can be used as a professional practice tool to track developmental trajectories (e.g., Cendan et al, 2015;Edwards, 2017;Keister et al, 2012;Lasater, 2007;Lyon, 2013). They have been used for supporting assessment for learning for those in medical professions (Lasater, 2007) and by students in most levels of education.…”