Evaluation of a Static Dashboard to Support Resident Learning and Competency Assessment

Vennemeyer, Scott; Parikh, Milan; Mu, Shuai; Kinnear, Benjamin; Wu, Danny T. Y.

doi:10.1109/vahc53729.2020.00012

Cited by 2 publications

(2 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The average scores were calculated for each of the two questions, then combined to produce a final rank for each component. A preliminary analysis of the ranking experiment has been published elsewhere, 36 but has been expanded upon greatly in this work.…”

Section: Discussionmentioning

confidence: 99%

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard

Vennemeyer,

Kinnear,

Gao

et al. 2023

Appl Clin Inform

View full text Add to dashboard Cite

Objectives Clinical Competency Committee (CCC) members employ varied approaches to the review process. This makes the design of a competency assessment dashboard that fits the needs of all members difficult. This work details a user-centered evaluation of a dashboard currently utilized by the Internal Medicine Clinical Competency Committee (IM CCC) at the University of Cincinnati College of Medicine and generated design recommendations. Methods Eleven members of the IM CCC participated in semistructured interviews with the research team. These interviews were recorded and transcribed for analysis. The three design research methods used in this study included process mapping (workflow diagrams), affinity diagramming, and a ranking experiment. Results Through affinity diagramming, the research team identified and organized opportunities for improvement about the current system expressed by study participants. These areas include a time-consuming preprocessing step, lack of integration of data from multiple sources, and different workflows for each step in the review process. Finally, the research team categorized nine dashboard components based on rankings provided by the participants. Conclusion We successfully conducted user-centered evaluation of an IM CCC dashboard and generated four recommendations. Programs should integrate quantitative and qualitative feedback, create multiple views to display these data based on user roles, work with designers to create a usable, interpretable dashboard, and develop a strong informatics pipeline to manage the system. To our knowledge, this type of user-centered evaluation has rarely been attempted in the medical education domain. Therefore, this study provides best practices for other residency programs to evaluate current competency assessment tools and to develop new ones.

show abstract

Section: Discussionmentioning

confidence: 99%

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard

Vennemeyer,

Kinnear,

Gao

et al. 2023

Appl Clin Inform

View full text Add to dashboard Cite

show abstract

“…Learning analytics and dashboards We use locally developed learning analytics 9 and visualization dashboards 12,26 in our CCC meetings and coaching discussions.…”

Section: Approach 3: Critical Epistemology Of Assessmentmentioning

confidence: 99%

Reconstructing Neurath’s Ship: A Case Study in Reevaluating Equity in a Program of Assessment

et al. 2023

Self Cite

View full text Add to dashboard Cite

Inequity in assessment has been described as a “wicked problem”—an issue with complex roots, inherent tensions, and unclear solutions. To address inequity, health professions educators must critically examine their implicit understandings of truth and knowledge (i.e., their epistemologies) with regard to educational assessment before jumping to solutions. The authors use the analogy of a ship (program of assessment) sailing on different seas (epistemologies) to describe their journey in seeking to improve equity in assessment. Should the education community repair the ship of assessment while sailing or should the ship be scrapped and built anew? The authors share a case study of a well-developed internal medicine residency program of assessment and describe efforts to evaluate and enable equity using various epistemological lenses. They first used a postpositivist lens to evaluate if the systems and strategies aligned with best practices, but found they did not capture important nuances of what equitable assessment entails. Next, they used a constructivist approach to improve stakeholder engagement, but found they still failed to question the inequitable assumptions inherent to their systems and strategies. Finally, they describe a shift to critical epistemologies, seeking to understand who experiences inequity and harm to dismantle inequitable systems and create better ones. The authors describe how each unique sea promoted different adaptations to their ship, and challenge programs to sail through new epistemological waters as a starting point for making their own ships more equitable.

show abstract

Evaluation of a Static Dashboard to Support Resident Learning and Competency Assessment

Cited by 2 publications

References 4 publications

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard

User-Centered Evaluation and Design Recommendations for an Internal Medicine Resident Competency Assessment Dashboard

Reconstructing Neurath’s Ship: A Case Study in Reevaluating Equity in a Program of Assessment

Contact Info

Product

Resources

About