How do clinical competency committees use different sources of data to assess residents’ performance on the internal medicine milestones?A mixed methods pilot study

Ekpenyong, Andem; Baker, Elizabeth A.; Harris, Ilene B.; Tekian, Ara; Abrams, Richard; Reddy, Shalini; Park, Yoon Soo

doi:10.1080/0142159x.2017.1353070

Cited by 52 publications

(45 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition to prompting us to reconsider what ‘counts’ as evidence, the results of our study have some practical implications. Given their intimate knowledge of assessment data, it has been speculated that CCCs are uniquely positioned to identify gaps or limitations in programmes of assessment . The results of this study highlight how this occurs in practice.…”

Section: Discussionmentioning

confidence: 84%

“…This process relies on the expertise and judgement of committee members and their ability to interpret performance evidence and transform it into a decision about trainee competence. The process of synthesising heterogeneous assessment data is fraught with challenges and the need for efficient, creative strategies to support the complex task of data aggregation has been communicated in the literature …”

Section: Introductionmentioning

confidence: 99%

“…Concerns around the fairness and transparency of progression decisions have been at the heart of this conversation . The current lack of systematic procedures for evidentiary review, weighting and interpretation and the heterogeneity of practices within committees have been problematised . In response, a variety of strategies have been proposed to standardise CCCs, including best operating practices, faculty training, data visualisation software and data analysis programs …”

Section: Introductionmentioning

confidence: 99%

“…A central thread of the scholarly discussion on CCCs has been focused on the quality of evidence produced for CCC review and has drawn attention to the issue of poor or variable data quality . Underpinning this conversation is an assumption that performance data generated for review can be objective – or that they can reliability communicate the ‘truth’ about workplace performance and competencies.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Some assembly required: tracing the interpretative work of Clinical Competency Committees

et al. 2019

View full text Add to dashboard Cite

Objectives This qualitative study describes the social processes of evidence interpretation employed by Clinical Competency Committees (CCCs), explicating how they interpret, grapple with and weigh assessment data. Methods Over 8 months, two researchers observed 10 CCC meetings across four postgraduate programmes at a Canadian medical school, spanning over 25 hours and 100 individual decisions. After each CCC meeting, a semi‐structured interview was conducted with one member. Following constructivist grounded theory methodology, data collection and inductive analysis were conducted iteratively. Results Members of the CCCs held an assumption that they would be presented with high‐quality assessment data that would enable them to make systematic and transparent decisions. This assumption was frequently challenged by the discovery of what we have termed ‘problematic evidence’ (evidence that CCC members struggled to meaningful interpret) within the catalogue of learner data. When CCCs were confronted with ‘problematic evidence’, they engaged in lengthy, effortful discussions aided by contextual data in order to make meaning of the evidence in question. This process of effortful discussion enabled CCCs to arrive at progression decisions that were informed by, rather than ignored, problematic evidence. Conclusions Small groups involved in the review of trainee assessment data should be prepared to encounter evidence that is uncertain, absent, incomplete, or otherwise difficult to interpret, and should openly discuss strategies for addressing these challenges. The answer to the problem of effortful processes of data interpretation and problematic evidence is not as simple as generating more data with strong psychometric properties. Rather, it involves grappling with the discrepancies between our interpretive frameworks and the inescapably subjective nature of assessment data and judgement.

show abstract

Section: Discussionmentioning

confidence: 84%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Some assembly required: tracing the interpretative work of Clinical Competency Committees

et al. 2019

View full text Add to dashboard Cite

show abstract

“…In the literature, these small groups are most often defined as summative decision‐making bodies, responsible for the review, collation and evaluation of trainee assessment data 1‐4 . To date, this body of literature has been organised around two themes: (a) developing best practices and forwarding practical advice for the assemblage and operation of CCs, 5‐9 and (b) understanding how the decision‐making work of the CC unfolds 1,10‐14 . The latter literature has a strong emphasis on summative decision making, explicating how these small groups collate and interpret aggregate assessment data to inform decisions about progression and entrustment 1,13,15,16 .…”

Section: Introductionmentioning

confidence: 99%

Beyond summative decision making: Illuminating the broader roles of competence committees

et al. 2020

View full text Add to dashboard Cite

Context: Although competence committees (CCs) are most often conceptualised as group decision-making bodies, policy documents forward a more ambitious vision and outline several additional roles for CCs that support the continuous quality improvement of education programmes and promote formative feedback. However, whether these functions are encompassed in the everyday work of CCs is currently unknown.Methods: This constructivist grounded theory study elaborates the range of roles taken up by CCs and illuminates the processes through which these roles are actualised. Two investigators observed 27 CC meetings (>80 hours) across seven diverse postgraduate programmes at a single Canadian institution. Following each observation, a semi-structured interview was conducted with one CC member. Data collection and analysis unfolded iteratively. Results:In this study, CCs did much more than make summative decisions about progression and entrustment; they identified a variety of problems that arose at the levels of both the individual learner and the programme, and leveraged their knowledge of assessment data to develop solutions. The problem-solving work of CCs was enabled by the in-depth data review, synthesis and analysis work that occurred before scheduled meetings, outside protected academic or administrative time. Although this work often appeared invisible to those outside the committee, the insights gleaned from data review provided committee members with essential information about how their programme of assessment was unfolding in practice. Conclusions:Competence committees may be an untapped resource that can support assessment for learning, local evaluation of competency-based medical education (CBME) implementation and continuous quality improvement for programmes of assessment. However, the ability of CCs to engage in this work is fragile and is currently sustained by the willingness of faculty members to devote their time and energy to it. The resourcing of CCs may have profound implications for translation of the theory of programmatic assessment and CBME into practice. 518 | PACK et Al. | INTRODUC TI ONCompetence committees (CCs) are a recent addition to postgraduate medical education. In the literature, these small groups are most often defined as summative decision-making bodies, responsible for the review, collation and evaluation of trainee assessment data. 1-4 To date, this body of literature has been organised around two themes: (a) developing best practices and forwarding practical advice for the assemblage and operation of CCs, 5-9 and (b) understanding how the decision-making work of the CC unfolds. 1,[10][11][12][13][14] The latter literature has a strong emphasis on summative decision making, explicating how these small groups collate and interpret aggregate assessment data to inform decisions about progression and entrustment. 1,13,15,16 Given the high-stakes nature of these summative decisions, it is unsurprising that they have been subject to significant scholarly interest. However, the focus ...

show abstract

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

et al. 2021

Self Cite

View full text Add to dashboard Cite

Key Points Question Can Milestones ratings be used to create reliable learning trajectories for resident physicians that identify developmental growth patterns toward unsupervised practice? Findings This cohort study of 3872 family medicine residents found that the Milestones assessment system had high reliability for measuring the developmental growth of learners. Each family medicine subcompetency included 3 or 4 distinct patterns of learning trajectories that could be used to support learner feedback. Meaning These findings suggest that identifying different patterns of learning trajectories using the Milestones assessment system could support and provide early remediation for struggling learners who may not meet graduation targets in training.

show abstract

How do clinical competency committees use different sources of data to assess residents’ performance on the internal medicine milestones?A mixed methods pilot study

Cited by 52 publications

References 15 publications

Some assembly required: tracing the interpretative work of Clinical Competency Committees

Some assembly required: tracing the interpretative work of Clinical Competency Committees

Beyond summative decision making: Illuminating the broader roles of competence committees

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

Contact Info

Product

Resources

About