Interrogating the Generalizability of Portfolio Assessments of Beginning Teachers: A Qualitative Study

Moss, Pamela; Sutherland, Lee‐Ann; Haniford, Laura; Miller, Renee; Johnson, Dale A.; Geist, Pamela Kaye; Koziol, Stephen M.; Star, Jon R.; Pecheone, Raymond

doi:10.14507/epaa.v12n32.2004

Cited by 9 publications

(10 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reliability of ratings was then formally assessed using Generalizability (G) Theory (Shavelson & Webb, 1991). G‐theory is particularly suitable as a framework for investigating the reliability of measures of instruction because it can assess the relative importance of multiple sources of error simultaneously (e.g., raters, tasks, occasions; see e.g., Moss et al, 2004). In the year 1 study each notebook was scored by multiple raters on each dimension; this is a crossed Teacher × Rater design with one facet of error (raters), which identifies three sources of score variance: true differences in instructional practice across teachers ($\sigma _{{\rm T}}^{{\rm 2}} $ ), mean differences between raters (i.e., variance in rater severity , $\sigma _{{\rm R}}^{{\rm 2}} $ ), and a term combining interaction and residual error ($\sigma _{{\rm TR,e}}^{2} $ ).…”

Section: Methodsmentioning

confidence: 99%

“…Systematically collected artifacts, assembled into portfolios or collected in other forms, can be used to measure various features of instructional practice, including some that are difficult to capture through surveys or observations (e.g., use of written feedback); moreover, because they contain direct evidence of classroom practice, artifacts are less susceptible to biases and social desirability effects. In addition to its potential for measuring instructional practice, the process of collecting artifacts can have value for teacher professional development (see e.g., Gerard, Spitulnik, & Linn, 2010; Moss et al, 2004). However, this method is not without limitations: collecting artifacts places a significant burden on teachers, who must save, copy, assemble, and even annotate and reflect on the materials.…”

Section: Features Of Instructional Practice In Middle School Sciencementioning

confidence: 99%

See 1 more Smart Citation

Measuring instructional practice in science using classroom artifacts: lessons learned from two validation studies

2011

View full text Add to dashboard Cite

With growing interest in the role of teachers as the key mediators between educational policies and outcomes, the importance of developing good measures of classroom processes has become increasingly apparent. Yet, collecting reliable and valid information about a construct as complex as instruction poses important conceptual and technical challenges. This article summarizes the results of two studies that investigated the properties of measures of instruction based on a teacher-generated instrument (the Scoop Notebook) that combines features of portfolios and self-report. Classroom artifacts and teacher reflections were collected from samples of middle school science classrooms and rated along 10 dimensions of science instruction derived from the National Science Education Standards; ratings based on direct classroom observations were used as comparison. The results suggest that instruments that combine artifacts and self-reports hold promise for measuring science instruction with reliability similar to, and sizeable correlations with, measures based on classroom observation. We discuss the implications and lessons learned from this work for the conceptualization, design, and use of artifact-based instruments for measuring instructional practice in different contexts and for different purposes. Artifact-based instruments may illuminate features of instruction not apparent even through direct classroom observation; moreover, the process of structured collection and reflection on artifacts may have value for professional development. However, their potential value and applicability on a larger scale depends on careful consideration of the match between the instrument and the model of instruction, the intended uses of the measures, and the aspects of classroom practice most amenable to reliable scoring through artifacts. We outline a research agenda for addressing unresolved questions and advancing theoretical and practical knowledge around the measurement of instructional practice. ß 2011 Wiley Periodicals, Inc. J Res Sci Teach 49: 2012 Keywords: science education; measurement of instruction; generalizability theory There is growing consensus among researchers and policymakers about the importance of accurate, valid, and efficient measures of instructional practice in science classrooms. Instruction directly or indirectly mediates the success of many school improvement efforts and thus accurate descriptions of what teachers do in classrooms as they attempt to implement reforms is key for understanding ''what works'' in education, and equally importantly, ''how?'' Many educational policies and programs rely on claims about the value of certain practices for improving student outcomes; for example, the No Child Left Behind legislation prompted schools to adopt scientifically based practices to improve the achievement of all students. Similarly, the reform teaching movement often recommends specific approaches to instruction designed to promote higher-level learning. More generally, the National Researc...

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Features Of Instructional Practice In Middle School Sciencementioning

confidence: 99%

Measuring instructional practice in science using classroom artifacts: lessons learned from two validation studies

2011

View full text Add to dashboard Cite

show abstract

“…Revealed also in this literature is the possibility that the certification examination may identify more than the usual numbers of false negatives and false positives, an issue that we think deserves much more research. There is a growing belief, based on recent research (Shutz and Moss et al, 2004), that the kinds of assessments used by the NBPTS gives only a brief glimpse of what a teacher is capable of under restrictions and controls. Typical, everyday classroom performance must necessarily differ from the performance displayed and judged from portfolio's and at assessment centers.…”

Section: Research On National Board Certified Teachersmentioning

confidence: 99%

National Board Certified Teachers andTheir Students' Achievement

Vandevoort

Amrein‐Beardsley

Berliner

2004

EPAA

View full text Add to dashboard Cite

Contemporary research on teaching indicates that teachers are powerful contributors to students' academic achievement, though the set and interrelationships of characteristics that make for high-quality and effective teaching have yet to be satisfactorily determined. Nevertheless, on the basis of the extant research and a vision of exemplary teaching, the National Board for Professional Teaching Standards stipulated a definition of a superior teacher. The Board did this without empirical evidence to support their claim that teachers' who meet the standards set by the Board were superior in promoting academic achievement to those who did not meet those standards. In the 17 years since the founding of the National Board, only a few empirical studies have addressed this important issue. In this study we compare the academic performance of students in the elementary classrooms of 35 National Board Certified teachers and their non-certified peers, in 14 Arizona school districts. Board Certified teachers and their principals provide additional information about these teachers and their schools. Four years of results from the Stanford Achievement Tests in reading, mathematics and language arts, in grades three Board Certified Teachers and Student Achievement 2 through six, were analyzed. In the 48 comparisons (four grades, four years of data, three measures of academic performance), using gain scores adjusted for students' entering ability, the students in the classes of National Board Certified Teachers surpassed students in the classrooms of non-Board certified teachers in almost threequarters of the comparisons. Almost one-third of these differences were statistically significant. In the cases where the students of non-Board certified teachers gained more in an academic year, none of the differences found were statistically significant. Effect size, translated into grade equivalents, informs us that the gains made by students of Board Certified teachers were over one month greater than the gains made by the students of non-Board certified peer teachers. Teachers identified through the assessments of the National Board for Professional Teaching Standards are, on average, more effective teachers in terms of academic achievement, one of the many outcomes of education for which teachers are responsible. This study does

show abstract

“…Most of the evidence-based research looking at these developments has focused on evaluating the quality of the evidence (Tillema, 2001), the need to ensure validity and reliability (Moss et al, 2004), and the technical infrastructure required to facilitate this migration (Gill, 2003). Beyond these, however, there are questions of control and ownership over the content of the e-portfolioissues of access to the data and questions of identity and privacy, on which the literature is less well defined.…”

Section: Advantages and Identified Risks Associated With E-portfoliosmentioning

confidence: 99%

E-Portfolios and Digital Identity: Some Issues for Discussion

McAlpine

2005

E-Learning and Digital Media

View full text Add to dashboard Cite

As awarding bodies modernise their procedures and incorporate elements of e-assessment into their qualifications, e-portfolios are emerging as a popular method of allowing candidates to display their abilities. All major United Kingdom awarding bodies now accept evidence from eportfolio products for at least some of their qualifications. Although there is a substantial body of emerging literature looking at how e-portfolios can be used and the practical implications of increasing provision, issues of identity and privacy with widespread use of e-portfolio products are rather less well explored. This article highlights the two dominant paradigms of e-portfolios -e-portfolio as assessment and e-portfolio as story -before exploring the concept of identity, particularly in relation to authentication, within an e-portfolio. It also considers the concept of 'emplotment' as defined by Ricoeur as a means of making sense of the narrative identity created. The article examines the implications and issues for awarding bodies associated with personal identity, privacy and surveillance which are raised by the widespread use of e-portfolios. It goes on to suggest some areas for further investigation and exploration.

show abstract

Interrogating the Generalizability of Portfolio Assessments of Beginning Teachers: A Qualitative Study

Cited by 9 publications

References 52 publications

Measuring instructional practice in science using classroom artifacts: lessons learned from two validation studies

Measuring instructional practice in science using classroom artifacts: lessons learned from two validation studies

National Board Certified Teachers andTheir Students' Achievement

E-Portfolios and Digital Identity: Some Issues for Discussion

Contact Info

Product

Resources

About