Program evaluation: An educator's portal into academic scholarship

Hosseini, Shera; Yılmaz, Yusuf; Shah, Kaushal; Gottlieb, Michael; Stehman, Christine R.; Hall, Andrew K.; Chan, Teresa M.

doi:10.1002/aet2.10745

Cited by 2 publications

(3 citation statements)

References 49 publications

(97 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Through multiple perspectives, and inspired by the proxies for work-as-done in the patient safety literature [ 14 ], this study presents proxies for training evaluation across three domains of design, experience and impact. SBE as a field has been searching for scholarly ways of exploring educational impact [ 13 ], of which this study presents an example. This study offers a framework to consider SBE impact beyond intended learning outcomes for the individual with particular calls to action heralded.…”

Section: Discussionmentioning

confidence: 99%

“…We must engage with theory to develop programme evaluation practices capable of capturing complexity [10,11]; SBE can be a particularly complex social intervention, for example through its propensity to evoke emotion [12]. We aim to answer the call for more scholarly approaches to programme evaluation [13], particularly within SBE, with our study.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Training as imagined? A critical realist analysis of Scotland’s internal medicine simulation programme

Kerins,

Ralston,

Stirling

et al. 2024

Adv Simul

View full text Add to dashboard Cite

Background Evaluating the impact of simulation-based education (SBE) has prioritised demonstrating a causal link to improved patient outcomes. Recent calls herald a move away from looking for causation to understanding ‘what else happened’. Inspired by Shorrock’s varieties of human work from patient safety literature, this study draws on the concept of work-as-done versus work-as-imagined. Applying this to SBE recognises that some training impacts will be unexpected, and the realities of training will never be quite as imagined. This study takes a critical realist stance to explore the experience and consequences, intended and unintended, of the internal medicine training (IMT) simulation programme in Scotland, to better understand ‘training-as-done’. Methods Critical realism accepts that there is a reality to uncover but acknowledges that our knowledge of reality is inevitably our construction and cannot be truly objective. The IMT simulation programme involves three courses over a 3-year period: a 3-day boot camp, a skills day and a 2-day registrar-ready course. Following ethical approval, interviews were conducted with trainees who had completed all courses, as well as faculty and stakeholders both immersed in and distant from course delivery. Interviews were audio-recorded, transcribed verbatim and analysed using critical realist analysis, influenced by Shorrock’s proxies for work-as-done. Results Between July and December 2023, 24 interviews were conducted with ten trainees, eight faculty members and six stakeholders. Data described proxies for training-as-done within three broad categories: design, experience and impact. Proxies for training design included training-as-prescribed, training-as-desired and training-as-prioritised which compete to produce training-as-standardised. Experience included training-as-anticipated with pre-simulation anxiety and training-as-unintended with the valued opportunity for social comparison as well as a sense of identity and social cohesion. The impact reached beyond the individual trainee with faculty development and inspiration for other training ventures. Conclusion Our findings highlight unintended consequences of SBE such as social comparison and feeling ‘valued as a trainee, valued as a person’. It sheds light on the fear of simulation, reinforcing the importance of psychological safety. A critical realist approach illuminated the ‘bigger picture’, revealing insights and underlying mechanisms that allow this study to present a new framework for conceptualising training evaluation.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Training as imagined? A critical realist analysis of Scotland’s internal medicine simulation programme

Kerins,

Ralston,

Stirling

et al. 2024

Adv Simul

View full text Add to dashboard Cite

show abstract

“…Although these approaches structure the steps of process improvement, a systematic review by Taylor et al found that many PDSA method studies failed to align with critical features (e.g., application of sequential, iterative cycles) [ 11 ]. Furthermore, medical educators frequently encounter competing responsibilities, limiting available time to dedicate to systematic PE, and may feel they need more training in PE approaches [ 12 ]. One surgical program noted the temptation to run on “autopilot” given the time and resource commitment required for program administration [ 13 ].…”

Section: Introductionmentioning

confidence: 99%

Picking Apart a Program Evaluation Committee: A Multiple Case Study Characterizing Primary Care Residency Program Evaluation Committee Structure, Program Improvement, and Outcomes

Lowry,

Merkebu,

Schall

et al. 2024

Cureus

View full text Add to dashboard Cite

Background: As of 2014, the Accreditation Council for Graduate Medical Education (ACGME) mandates initiating a Program Evaluation Committee (PEC) to guide ongoing program improvement. However, little guidance nor published reports exist about how individual PECs have undertaken this mandate. Objective: To explore how four primary care residency PECs configure their committees, review program goals and undertake program evaluation and improvement. Methods: We conducted a multiple case study between December 2022 and April 2023 of four purposively selected primary care residencies (e.g., family medicine, pediatrics, internal medicine). Data sources included semi-structured interviews with four PEC members per program and diverse program artifacts. Using a constructivist approach, we utilized qualitative coding to analyze participant interviews and content analysis for program artifacts. We then used coded transcripts and artifacts to construct logic models for each program guided by a systems theory lens. Results: Programs adapt their PEC structure, execution, and outcomes to meet short- and long-term needs based on organizational and program-unique factors such as size and local practices. They relied on multiple data sources and sought diverse stakeholder participation to complete program evaluation and improvement. Identified deficiencies were often categorized as internal versus external to delineate PEC responsibility, boundaries, and feasibility of interventions. Conclusion: The broad guidance provided by the ACGME for PEC configuration allows programs to adapt the committee based on individual needs. However, further instruction on program evaluation and organizational change principles would augment existing PEC efforts.

show abstract

Program evaluation: An educator's portal into academic scholarship

Cited by 2 publications

References 49 publications

Training as imagined? A critical realist analysis of Scotland’s internal medicine simulation programme

Training as imagined? A critical realist analysis of Scotland’s internal medicine simulation programme

Picking Apart a Program Evaluation Committee: A Multiple Case Study Characterizing Primary Care Residency Program Evaluation Committee Structure, Program Improvement, and Outcomes

Contact Info

Product

Resources

About