Psychometric Properties and Instructional Utility of Assessing Special Education Teacher Candidate Knowledge with Portfolios

Abstract

Standardized portfolios are being used by teachers and teacher candidates to demonstrate subject matter knowledge for certification or to be considered highly qualified; however, the psychometric adequacy of data used for this purpose has not been evaluated. In the current study, we examined the interscorer reliability, concurrent, predictive, and discriminate validity, and instructional utility of an assessment portfolio used in a special education teacher education program. The estimate of interscorer reliability was .91, validity coefficients ranged from .53 to .60, and the discriminate validity coefficient was -.01. A comparison of scores for instructional objectives suggested relative difficulties writing adequate short-term objectives, which was then further emphasized in subsequent instruction and resulted in increased scores. Suggestions for practice and future research are included.

Reframing Portfolio Evidence Empowering Teachers through Single-Case Frameworks

“…Despite claims of increased skills development, facilitated reflection, and professional development, few implementations document how portfolios influence these outcomes. After decades of portfolio use among pre and in-service teachers and years of research criticizing lack of validity and reliability in portfolio claims (Burns & Haight, 2005;Tillema & Smith, 2007;Zeichner & Wray, 2001), few researchers have considered how to select, organize, and examine artifacts to provide valid measures of practice or document them in such a way that they may be replicated by others. This article provides a framework for artifact collection that attempts to bridge previous limitations by relying on flexible foci, validated methods, data-driven decisions, and documentation of problem solving processes for purposes of professional development.…”

“…When teachers and mentors reflect on memories to examine practice, they may misrepresent events, misinterpret included artifacts, and misdirect inquiry and assessment. Although Naizer (1997) and Burns and Haight (2005) found that portfolio success was a good indicator of course grades, strategic and pedagogical knowledge, and prior teaching experience, Tillema and Smith (2007) concluded that portfolio assessment was often confounded by assessor expectations regardless of program objectives and grading criteria. It is questionable whether principles of evidence credibility and relevance, instrumental to event reconstruction and argument formation, are adequately taught to teachers during portfolio practices.…”

Development and Testing of a Portfolio Evaluation Scoring Tool

“…Regardless of academic discipline, faculty are challenged to create portfolio evaluation tools (PETs) that are easy to understand and implement, promote objective analysis, and result in scores that provide a meaningful evaluation of student performance specific to the purposes of the portfolio. The literature includes several articles that describe the experiences of faculty attempting to implement portfolios for course or program evaluation within nursing (Cook, Kase, Middelton, & Monsen, 2003;Jasper & Fulton, 2005;Kear & Bear, 2007;Martin, Kinnick, Hummel, Clukey, & Baird, 1997;Ruholl, 2000;Wenzel et al, 1998) and in other academic disciplines (Annis & Jones, 1995;Burch, 1997;Burns & Haight, 2005;Mac-Donald, 1997;Morgan, 1999). These studies usually focus on the content of portfolios; rarely are the processes used to judge the quality of materials contained in portfolios explained in any detail.…”

Validity Evidence of an Electronic Portfolio for Preservice Teachers

“…Although national professional organizations have established content‐specific teacher competency standards, there is little consensus about the best way to measure the preservice teacher's essential knowledge, skills, and dispositions that will result in P‐12 classroom success. Many teacher education programs have developed or adopted strategies for assessing the competencies of its preservice teachers often without examining the reliability or validity of those measures (Burns & Haight, 2005).…”

“…In one of the few studies on portfolio validity in teacher education, Burns and Haight (2005) examined a portfolio designed for an undergraduate special education assessment course. The concurrent validity was examined by correlating the portfolio scores with results of quizzes and papers assigned to the students in the course.…”