This article develops a validity argument approach for use on observation protocols currently used to assess teacher quality for high-stakes personnel and professional development decisions. After defining the teaching quality domain, we articulate an interpretive argument for observation protocols. To illustrate the types of evidence that might compose a validity argument, we draw on data from a validity study of the Classroom Assessment Scoring System for secondary classrooms. Based on data from 82 Algebra classrooms, we illustrate how data from observation scores, valueadded models, generalizability studies, and measures of teacher knowledge, student achievement, and teacher and student beliefs could be used to build a validity argument for observation protocols. Strengths and limitations of the validity argument approach as well as the issues the approach raises for observation protocol validity research are considered.Recent federal legislation has put states and districts under unprecedented pressure to improve teaching quality through evaluation (United States Department of Education, 2009; United States Department of Education Office of Planning Evaluation and Policy Development, 2010).

Frameworks that seek to understand how knowledge restructuring occurs and how to build a learning environment that facilitates this restructuring raise important philosophical, psychological and pedagogical questions and issues about how conceptual change occurs and what characteristics of knowledge growth ought to be a part of curricula and learning environments. Implicit in emphasizing the how is a shift in science educations' perspective from one that embraces “scientists' ways of knowing” as the dominant objective towards one that favors “positioning the learner for the next step.” This change in perspective and approach represents a radical and complex departure from common practice. This article advances a piecemeal model of the character and mechanism of restructuring and then describes a model of educational practice designed to facilitate this form of restructuring. We argue that a piecemeal developmental perspective of conceptual change would offer quite different criteria for deciding what to teach and how to teach. The adoption of conceptual change teaching models implies teacher empowerment of a kind we have yet to fully understand. Empowering teachers with appropriate philosophical and psychological models for the selection and the sequencing of instructional tasks would aid in their describing and prescribing effective or meaningful learning strategies. Central to this educational model is a broadened and integrated view of assessment and instruction that we are calling a portfolio culture. The essential characteristic of this culture is that it creates opportunities for teachers and students to confront and develop their scientific understanding and to equip students with the tools necessary to take increased responsibility for their own restructuring, to assess for themselves what might be the next steps.

No abstract

No abstract

The Educative Teacher Performance Assessment (edTPA) is a system of standardized portfolio assessments of teaching performance mandated for use by educator preparation programs in 18 states, and approved in 21 others, as part of initial certification for preservice teachers. Because of the high stakes involved for examinees, it is critical that the scores produced and resulting decisions are meaningful and meet robust standards of validity and technical quality for educational measurements. We examined the technical documentation of edTPA and raise serious concerns about scoring design, the reliability of the assessments, and the consequential impact on decisions about edTPA candidates. In light of these findings, we argue that the proposed and actual uses of the edTPA are currently unwarranted on technical grounds.

Classroom observation of teachers is a significant part of educational measurement; measurements of teacher practice are being used in teacher evaluation systems across the country. This research investigated whether observations made live in the classroom and from video recording of the same lessons yielded similar inferences about teaching. Using scores on the Classroom Assessment Scoring SystemSecondary (CLASS-S) from 82 algebra classrooms, we explored the effect of observation mode on inferences about the level or ranking of teaching in a single lesson or in a classroom for a year. We estimated the correlation between scores from the two observation modes and tested for mode differences in the distribution of scores, the sources of variance in scores, and the reliability of scores using generalizability and decision studies for the latter comparisons. Inferences about teaching in a classroom for a year were relatively insensitive to observation mode. However, time trends in the raters' use of the score scale were significant for two CLASS-S domains, leading to mode differences in the reliability and inferences drawn from individual lessons. Implications for different modes of classroom observation with the CLASS-S are discussed.As the education reform movement increasingly focuses on teachers and teaching, educators, policymakers, and researchers need valid and reliable measures of teaching that can be used to evaluate individual teachers, provide guidance for improving teaching performance, and support research in ways that advance instruction and classroom dialog and practice. Nearly 20 years ago, Jaeger (1993) identified mode of observation as potentially contributing to the psychometric properties of measuring teaching, but little research on mode effects has occurred since. Renewed interest in measuring teaching and the large-scale use of observations for teacher evaluation systems has raised questions about the affordances of video capture, heightening the need for information on the comparability of scoring video and live observations. We present the first large-scale comparison of observation mode in the assessment of mathematics teaching.Observations of teaching are viewed as very useful data sources about teaching quality because they provide assessments that incorporate not only observation of the teacher's teaching but also the level of student engagement, the cognitive complexity of student-teacher interactions, and the subject matter focus and depth of instruction (Erickson, 2006;Jaeger, 1993). Video recording of classrooms is an alternative with practical advantages (Brunvard, 2010), especially with recent technological advances in the capture and transmission of digital audio and video.Extant research has shown very little difference in scores resulting from video and live observation. However, the studies were either not based on any rigorous evaluation (and were therefore inconclusive) or were conducted on data for nonclassroom contexts. Frederiksen, Sipusic, Gamoran, and Wolfe (1992) found ...

No abstract

Valid measurement of how students’ experiences in secondary school classrooms lead to gains in learning requires a developmental approach to conceptualizing classroom processes. This article presents a potentially useful theoretical model, the Teaching Through Interactions framework, which posits teacher-student interactions as a central driver for student learning and that teacher-student interactions can be organized into three major domains. Results from 1,482 classrooms provide evidence for distinct emotional, organizational, and instructional domains of teacher-student interaction. It also appears that a three-factor structure is a better fit to observational data than alternative one- and two-domain models of teacher-student classroom interactions, and that the three-domain structure is generalizable from 6th through 12th grade. Implications for practitioners, stakeholders, and researchers are discussed.