Sources of invalidity in measuring classroom behavior

Abstract

The concepts of teacher effectiveness and teacher evaluation are growing in importance. Attempts to identify the classroom behavior of more and less effective teachers and evaluate the extent to which they reach pre-determined criteria of effectiveness have led to a heterogeneous array of statistical tools and measurement paradigms, many of which contain sources of invalidity. This paper (a) critically reviews past attempts to identify the behavioral correlates of effective teaching; (b)poses a systematic measurement framework for more effectively studying classroom behavior, and (c) identifies specific sources of invalidity in measuring the behaviors within this framework. This paper is a review of the methodological problems uncovered by relatively recent efforts in the U.S.A. to study the nature of teacher effectiveness and to evaluate the performance of individual teachers. The former concept -teacher effectiveness -derives from almost two decades of research, conducted in this country and elsewhere, to identify the behavioral correlates of "more effective" and "less effective" teachers. The latter concept -teacher evaluation -stems from comparatively recent efforts to design and implement schemes for appraising individual teachers, a practice stimulated primarily by an increasing number of state and local mandates requiring annual, systematic evaluation of elementary and secondary school teachers.

Cognitive Style and Advance Organizers in Learning and Retention

“…This technique, though preferred by Overall and Woodward (1977) and Borich (1977), would lead, therefore, to the removal of variance correctly attributable to the effects of the treatments. Two-way analysis of variance of the residual gain scores was, therefore, carried out, their distribution having been shown to be approximately normal by Geary's test (D 'Agostino, 1970).…”

“…Analysis of covariance, though widely regarded by many as a suitable technique for data of this type to adjust for initial differences in learning scores, could not be applied to investigate the effects of treatment on retention scores since the covariate, having been obtained after application of the treatments, is correlated with them. This technique, though preferred by Overall and Woodward (1977) and Borich (1977), would lead, therefore, to the removal of variance correctly attributable to the effects of the treatments. Two-way analysis of variance of the residual gain scores was, therefore, carried out, their distribution having been shown to be approximately normal by Geary's test (D'Agostino, 1970).…”

Factorial Analysis of Generalizability

“…Reviews (Borich, 1977;Shavelson and Atwood, 1977) and psychological investigations of teacher effectiveness (Erlich and Shavelson, 1976a; Erlich and Borich, 1976a, b;Evertson, Anderson, Edgar, Minter and Brophy, 1977) have shown that one explanation for past failures to find empirically consistent relationships between classroom interactions and student achievement may lie in the process employed for quantifying both process and product variables. The purpose of this paper was to delineate some measurement problems raised by classical test theory, to describe how generalizability theory may be a solution to these problems, and to present a computer program for studying reliability (generalizability) with a 4 factor design.…”

Assessing the stability of teacher behavior.

“…Despite the many extensive studies of teacher behavior, affirmative answers to this question have been rare. For example, Borich (1977) stated that "the results of these studies suggest that teacher behavior may be unstable across long periods of time and content" (p. 300). l Moreover, researchers lack confidence in the results of these research efforts; the conclusion of Shavelson and…”