Avoid reinventing the wheel: implementation of the Ottawa Clinic Assessment Tool (OCAT) in Internal Medicine

Abstract

BackgroundWorkplace based assessment (WBA) is crucial to competency-based education. The majority of healthcare is delivered in the ambulatory setting making the ability to run an entire clinic a crucial core competency for Internal Medicine (IM) trainees. Current WBA tools used in IM do not allow a thorough assessment of this skill. Further, most tools are not aligned with the way clinical assessors conceptualize performances. To address this, many tools aligned with entrustment decisions have recently been published. The Ottawa Clinic Assessment Tool (OCAT) is an entrustment-aligned tool that allows for such an assessment but was developed in the surgical setting and it is not known if it can perform well in an entirely different context. The aim of this study was to implement the OCAT in an IM program and collect psychometric data in this different setting. Using one tool across multiple contexts may reduce the need for tool development and ensure that tools used have proper psychometric data to support them.MethodsPsychometrics characteristics were determined. Descriptive statistics and effect sizes were calculated. Scores were compared between levels of training (juniors (PGY1), seniors (PGY2s and PGY3s) & fellows (PGY4s and PGY5s)) using a one-way ANOVA. Safety for independent practice was analyzed with a dichotomous score. Variance components were generated and used to estimate the reliability of the OCAT.ResultsThree hundred ninety OCATs were completed over 52 weeks by 86 physicians assessing 44 residents. The range of ratings varied from 2 (I had to talk them through) to 5 (I did not need to be there) for most items. Mean scores differed significantly by training level (p < .001) with juniors having lower ratings (M = 3.80 (out of 5), SD = 0.49) than seniors (M = 4.22, SD = − 0.47) who had lower ratings than fellows (4.70, SD = 0.36). Trainees deemed safe to run the clinic independently had significantly higher mean scores than those deemed not safe (p < .001). The generalizability coefficient that corresponds to internal consistency is 0.92.ConclusionsThis study’s psychometric data demonstrates that we can reliably use the OCAT in IM. We support assessing existing tools within different contexts rather than continuous developing discipline-specific instruments.Electronic supplementary materialThe online version of this article (10.1186/s12909-018-1327-7) contains supplementary material, which is available to authorized users.

Workplace‐based assessments in postgraduate medical education: A hermeneutic review

“…We identified one study considering the economic properties of WBAs, which estimated large time (and therefore financial) costs associated with the implementation of a WBA system. 38 Further feasibility themes concerned prohibitive time requirements, particularly for some technological platforms, and that high numbers of WBAs are required to produce reliable judgements, both of which impede using WBAs in busy workloads. 3,9,10,13,[31][32][33]36,37,39,[42][43][44][45][46][47][48]…”

“…27 Conversely, others believe assessment should be tailored to the needs of the trainee to maximise its utility. 27 Most studies examining WBAs' psychometric properties have used generalisability theory, 26,38,39,43,46,53 although one used classic reliability theories. 37 Alternatively, some argue that the centrality of context requires using a social constructivist epistemology.…”

“…37 Alternatively, some argue that the centrality of context requires using a social constructivist epistemology. 27,28 Similarly, although some papers used either Kane's 11 or Messick's 54 validity frameworks, 10,38,43 others have argued that an interactional framework should be used to account for how the interactions between a WBA's aspects (eg, trainee, assessor, tool, environment, case, etc.) affect the outcome.…”

“…3,27,40,41,45 Additionally, the assessor's level of self-reflexivity may enhance their observational capacity, 50 whereas assessors believe junior trainees are more engaged and receptive to feedback than senior trainees. 42 The definitions, frameworks and scales in the tool can guide assessors' observations and, in turn, decision making, 13,23,29,30,38,42,44,45,48,52 whereas training organisation policies such as requirements regarding minimum quantities or frequencies of WBAs may stimulate inactive trainers and trainees. 41 Perhaps the most potent determinants are how these factors interact.…”

Entrustment Ratings in Internal Medicine Training: Capturing Meaningful Supervision Decisions or Just Another Rating?

“…Initial evidence demonstrates that residents' scores improve over time, both within individuals and across training levels. [21][22][23] However, given internal medicine's long familiarity with workplace-based assessments such as the mini-cex, faculty may be using a 5-point rating scale as they always have, irrespective of the construct that is being assessed. We need more robust and diverse validity evidence to determine if entrustment is truly what is being measured.…”