The reliability and validity of a portfolio designed as a programmatic assessment of performance in an integrated clinical placement

Abstract

BackgroundLittle is known about the technical adequacy of portfolios in reporting multiple complex academic and performance-based assessments. We explored, first, the influencing factors on the precision of scoring within a programmatic assessment of student learning outcomes within an integrated clinical placement. Second, the degree to which validity evidence supported interpretation of student scores.MethodsWithin generalisability theory, we estimated the contribution that each wanted factor (i.e. student capability) and unwanted factors (e.g. the impact of assessors) made to the variation in portfolio task scores. Relative and absolute standard errors of measurement provided a confidence interval around a pre-determined pass/fail standard for all six tasks. Validity evidence was sought through demonstrating the internal consistency of the portfolio and exploring the relationship of student scores with clinical experience.ResultsThe mean portfolio mark for 257 students, across 372 raters, based on six tasks, was 75.56 (SD, 6.68). For a single student on one assessment task, 11% of the variance in scores was due to true differences in student capability. The most significant interaction was context specificity (49%), the tendency for one student to engage with one task and not engage with another task. Rater subjectivity was 29%. An absolute standard error of measurement of 4.74%, gave a 95% CI of +/- 9.30%, and a 68% CI of +/- 4.74% around a pass/fail score of 57%. Construct validity was supported by demonstration of an assessment framework, the internal consistency of the portfolio tasks, and higher scores for students who did the clinical placement later in the academic year.ConclusionA portfolio designed as a programmatic assessment of an integrated clinical placement has sufficient evidence of validity to support a specific interpretation of student scores around passing a clinical placement. It has modest precision in assessing students’ achievement of a competency standard. There were identifiable areas for reducing measurement error and providing more certainty around decision-making. Reducing the measurement error would require engaging with the student body on the value of the tasks, more focussed academic and clinical supervisor training, and revisiting the rubric of the assessment in the light of feedback.

Deconstructing programmatic assessment

“…Programmatic assessment has been described in medical programs, 13 – 17 postgraduate residency programs, 18 veterinary courses, 19 and nutrition and dietetics. 20 Developed and implemented mostly in the Netherlands, 16 , 17 , 19 , 21 it has also been implemented in the USA, 14 , 15 Canada, 18 Australia, 20 , 22 and New Zealand. 13 …”

“… 3 Some descriptions of programmatic assessment focus primarily on the use of a portfolio as a means of capturing the evidence and contributing to decision making. 14 , 15 , 22 However, it may be timely to consider (and encourage) other models. To do this requires us to dissect out the key elements, which we suggest are: Create clear expectations of required learning Undertake purposeful selection of assessments Focus on those learners who need extra attention and/or extra information Separate data from decisions Aggregate by attribute, not method or timing Make decisions on aggregate, not on individual assessments Promote sharing of information and dialogue around narrative rather than numbers Maximize the assessments to guide learning …”

Rigour in moderation processes is more important than the choice of method

“…It has been suggested that inter-assessor reliability (that is, the consistency in mark between assessors) for high-stakes medical education assessments should show correlations of r= 0.70-0.80 (Roberts, Shadbolt, Clark, & Simpson, 2014). Average inter-assessor reliability is in fact typically ~0.60, is affected by a wide range of factors, and will vary across submissions (e.g.…”

“…Average inter-assessor reliability is in fact typically ~0.60, is affected by a wide range of factors, and will vary across submissions (e.g. Bloxham & Price, 2013, Elton & Johnston, 2002Roberts et al, 2014).…”

Exploring the Role of Self-Motives in Postgraduate Trainees’ Feedback-Seeking Behavior in the Clinical Workplace: A Multicenter Study of Workplace-Based Assessments From the United Kingdom

“… HPAC's programmatic assessment case study 2016 the portfolio approach to a curriculum module described in Roberts et al (2014)  the approach developed by the Association of American Medical Colleges 2014which is concerned with a learner's trustworthiness to complete clinical tasks that integrate different skills and knowledge independently (ten Cate 2013).…”