Summative assessment of portfolios: an examination of different approaches to agreement over outcomes

Abstract

The issue of arriving at agreement over outcomes in summative assessment of portfolios has been a major concern, given the complexity of the assessment task, the educational and political context, and the widespread and growing use of portfolios in higher education. This article examines research findings in this area. The discussion takes place in a philosophical and theoretical context. The first section of this article considers various approaches to portfolio assessment (e.g. positivist, interpretivist, feminist) and the assumptions underlying them. The second section examines research findings in these different traditions, pointing out the findings and gaps in each, as well as suggesting potential meeting points between them. The article does not argue that any approach offers all answers to all problems connected with agreement over outcomes in portfolio summative assessment. Rather, the purpose of this article is to clarify the choices facing assessors. The underlying issues raised here have relevance to other methods of assessment, apart from portfolios.

Use of Reflective Portfolios in Health Sciences Education

“…39,[42][43]46,48,52 While inter-rater reliability has been suggested as the goal of reliability assessment, the inherent qualitative nature of reflective portfolios may preclude achieving acceptable reliability estimates, especially in regard to summative estimates. Driessen et al suggested that while having stricter assessment criteria and increased structure to the reflective portfolio process would increase inter-rater reliability, it would also negate the proposed benefits of the reflective portfolio itself.…”

The Market-Oriented Management Style of Private Vocational Schools in Complying with the Labor Market in Southern Thailand

“…A variable of the establishment includes the process of evaluation, which reveals significant differences between the teacher and developers. Sometimes, the evaluation should be scheduled the training for evaluation (Kaslow et al, 2007;Kane, 2008;Johnston, 2004). Variables with educational establishments are of paramount importance in two sectors, which cooperate together.…”

Assessors’ approaches to portfolio assessment in Assessment of Prior Learning procedures

“…This interview consisted of open-ended questions about the following topics: the deciding factors in evaluating a portfolio, the fairness of portfolio assessment alone, the extent to which the portfolios meet the rating criteria and the whole decisionmaking process (based on the aspects of Johnston [2004]: procedure to reach the final decision, level of judgement, assessment method, cooperation and agreement between assessors, assessor types, use of scales [holistic or elementary] and criteria, moment for final decision-making). To illustrate their responses with examples from the portfolios, each assessor had two assessed portfolios at hand; one with additional assessments and one without.…”

“…However, studies by Van der Schaaf and Stokking (2008) and Baume, Yorke, and Coffey (2004) show that assessors are capable of articulating the reasons behind their judgements, which helps us gain more insight into decision-making in portfolio assessment. Johnston (2004) mentions several aspects influencing the decision-making process. First, the way a final grade is calculated.…”

“…Johnston (2004) suggests using reliability and validity within an interpretative approach, which allows discussion of values and standards, and bridging between the local context and the curriculum. Negotiating with co-assessors over the outcomes will improve the reliability and validity.…”