Developing a dashboard to meet Competence Committee needs: a design-based research project

Thoma, Brent; Bandi, Venkat; Carey, Robert F.; Mondal, Debajyoti; Woods, Robert A.; Martin, Lynsey J; Chan, Teresa M.

doi:10.36834/cmej.68903

Cited by 28 publications

(60 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…16,26 Whereas traditional assessment occurred within discrete courses or clinical rotations, learning analytics make it possible to perform sophisticated analyses of a learner's developmental trajectory over time and across programs. [27][28][29] In addition, programs can use learning analytics data predictively to identify risks for poor performance and intervene to change a learner's trajectory. 7,30,31 Learner assessment and program evaluation in medical education draw on models and theories that are well described in the literature.…”

Section: Discussionmentioning

confidence: 99%

Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus From an International Cohort of Medical Educators

Thoma

Warm

Hamstra

et al. 2020

Journal of Graduate Medical Education

Self Cite

View full text Add to dashboard Cite

Background With the implementation of competency-based assessment systems, education programs are collecting increasing amounts of data about medical learners. However, learning analytics are rarely employed to use this data to improve medical education. Objective We identified outstanding issues that are limiting the effective adoption of learning analytics in medical education. Methods Participants at an international summit on learning analytics in medical education generated key questions that need to be addressed to move the field forward. Small groups formulated questions related to data stewardship, learner perspectives, and program perspectives. Three investigators conducted an inductive qualitative content analysis on the participant questions, coding the data by consensus and organizing it into themes. One investigator used the themes to formulate representative questions that were refined by the other investigators. Results Sixty-seven participants from 6 countries submitted 195 questions. From them, we identified 3 major themes: implementation challenges (related to changing current practices to collect data and utilize learning analytics); data (related to data collection, security, governance, access, and analysis); and outcomes (related to the use of learning analytics for assessing learners and faculty as well as evaluating programs and systems). We present the representative questions and their implications. Conclusions Our analysis highlights themes regarding implementation, data management, and outcomes related to the use of learning analytics in medical education. These results can be used as a framework to guide stakeholder education, research, and policy development that delineates the benefits and challenges of using learning analytics in medical education.

show abstract

Section: Discussionmentioning

confidence: 99%

Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus From an International Cohort of Medical Educators

Thoma

Warm

Hamstra

et al. 2020

Journal of Graduate Medical Education

Self Cite

View full text Add to dashboard Cite

show abstract

“…Well-defined milestones consist of observable markers of a resident's ability along a developmental continuum that can be displayed in dashboard format for feedback. 12 The ACGME Milestones project with its planned updates for July 2020 was developed to similarly align the core clinical competencies in radiology education to this end. 13…”

Section: Discussionmentioning

confidence: 99%

Quantifying Impact of Disruption to Radiology Education During the COVID-19 Pandemic and Implications for Future Training

Shi

Miskin

Dabiri

et al. 2021

Current Problems in Diagnostic Radiology

View full text Add to dashboard Cite

Purpose The aim of this study was to assess the impact on radiology resident education due to the COVID-19 pandemic in order to inform future educational planning. Methods During a 10-week study period from March 16 to May 22, 2020, changes to educational block-weeks (BW) of first through fourth year residents (R1-4) were documented as disrupted in the setting of the COVID-19 pandemic. The first 5 weeks and the second 5 weeks were evaluated separately for temporal differences. Overall and mean disrupted BW per resident were documented. Wilcoxon rank-sum tests were used to assess pairwise differences between classes with Bonferroni-adjusted P -values, as well as differences in the early versus later phase of the pandemic. Results Of 373 BW, 56.6% were assigned to virtual curriculum, 39.4% radiology clinical duties, 2.9% illness, and 1.1% reassignment. Scheduling intervention affected 6.2 ± 2.3 (range 1-10) mean BW per resident over the 10-week study period. The R3 class experienced the largest disruption, greater than the R2 classes, and statistically significantly more than the R1 and R4 classes (both P < 0.05). The second half of the pandemic caused statistically significantly more schedule disruptions than the first half ( P = 0.009). Discussion The impact of COVID-19 pandemic varied by residency class year, with the largest disruption of the R3 class and the least disruption of the R4 class. To optimize future educational opportunities, shifting to a competency-based education paradigm may help to achieve proficiency without extending the length of the training program.

show abstract

“…Building on our previous work exploring the needs of faculty members who serve on CBD competence committees, 31 we utilized a design-based research method to identify resident needs for data, analytics, and visualizations of their assessment data and created a dashboard to facilitate resident learning. This paper utilized design-based research to identify and address resident needs in a competency-based medical education program.…”

Section: Discussionmentioning

confidence: 99%

“…The Design-Based Research methodology (11,24) that we employed aligned with the approach outlined in our previous work on competency committee dashboards. 31 Design-based research is an "authentic, contextually aware, collaborative, theoretically focused, methodologically diverse, practical, iterative, and operation-oriented" process 11,25 which aims to bridge practice and research in education through the integration of investigation and intervention. 24,25,27 Our process consisted of the four phases of design-based research.…”

Section: Design Based Research Processmentioning

confidence: 99%

Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project

et al. 2020

Self Cite

View full text Add to dashboard Cite

Background: Canadian specialty programs are implementing Competence By Design, a competency-based medical education (CBME) program which requires frequent assessments of entrustable professional activities. To be used for learning, the large amount of assessment data needs to be interpreted by residents, but little work has been done to determine how visualizing and interacting with this data can be supported. Within the University of Saskatchewan emergency medicine residency program, we sought to determine how our residents’ CBME assessment data should be presented to support their learning and to develop a dashboard that meets our residents’ needs. Methods: We utilized a design-based research process to identify and address resident needs surrounding the presentation of their assessment data. Data was collected within the emergency medicine residency program at the University of Saskatchewan via four resident focus groups held over 10 months. Focus group discussions were analyzed using a grounded theory approach to identify resident needs. This guided the development of a dashboard which contained elements (data, analytics, and visualizations) that support their interpretation of the data. The identified needs are described using quotes from the focus groups as well as visualizations of the dashboard elements. Results: Resident needs were classified under three themes: (1) Provide guidance through the assessment program, (2) Present workplace-based assessment data, and (3) Present other assessment data. Seventeen dashboard elements were designed to address these needs. Conclusions: Our design-based research process identified resident needs and developed dashboard elements to meet them. This work will inform the creation and evolution of CBME assessment dashboards designed to support resident learning.

show abstract

Developing a dashboard to meet Competence Committee needs: a design-based research project

Cited by 28 publications

References 37 publications

Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus From an International Cohort of Medical Educators

Next Steps in the Implementation of Learning Analytics in Medical Education: Consensus From an International Cohort of Medical Educators

Quantifying Impact of Disruption to Radiology Education During the COVID-19 Pandemic and Implications for Future Training

Developing a dashboard to meet the needs of residents in a competency-based training program: A design-based research project

Contact Info

Product

Resources

About