An Investigation of Alternative Methods for Item Mapping in the National Assessment of Educational Progress

Zwick, Rebecca; Senturk, Deniz; Wang, Joyce; Loomis, Susan Cooper

doi:10.1111/j.1745-3992.2001.tb00059.x

Cited by 39 publications

(35 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…One intuitive and interpretable way of presenting student scores is through item mapping (Ebel, 1962;Masters, Adams, & Lokan, 1994;Huynh, 1998), which allows a deeper interpretation of scores with respect to specific item types and item contents, and it is implemented in large-scale assessment programs such as the National Assessment of Educational Progress (NAEP) and PISA (Zwick, Senturk, Wang, & Loomis, 2001;OECD, 2009). Mapping allows the description of items a student is less or more likely to endorse, and it is also possible to infer evidence on particular groups of examinees.…”

Section: Item Mappingmentioning

confidence: 99%

Gender differences in performance in mathematics at the end of lower secondary school in Italy

Matteucci

Mignani

2011

Learning and Individual Differences

View full text Add to dashboard Cite

Section: Item Mappingmentioning

confidence: 99%

Gender differences in performance in mathematics at the end of lower secondary school in Italy

Matteucci

Mignani

2011

Learning and Individual Differences

View full text Add to dashboard Cite

“…einzig item 19 (im Unterricht werden überfachliche Zusammenhänge aufgezeigt.) und item 23 (Die Schüler/innen evaluation von Unterrichtsstandards Stone et al 1999;Zwick et al 2001) der Analysedaten, das wie folgt zu lesen ist: Jede Ausprägung ("trifft nicht zu" bis "trifft zu") der 30 eingesetzten items (zu erkennen an der itemnummer 1 bis 30 mit dem Zusatz .2 für "trifft eher nicht zu", .3 für "trifft eher zu" und .4 für "trifft zu" in der Spalte itemschwierigkeit) hat eine spezifische Auftretenswahrscheinlichkeit, die zwischen den Items konstant gehalten wird und aufgrund der gleichzeitigen Darstellung von itemschwierigkeit und Qualität des Unterrichts auf einer gemeinsamen Skala, mit einen mittelwert von null und einer Standardabweichung von eins, ein indikator für das Auftreten einer bestimmten Unterrichtsqualitätsausprägung ist.…”

Section: Dimensionalitätunclassified

Evaluation von Unterrichtsstandards

Pietsch¹

2010

Z Erziehungswiss

View full text Add to dashboard Cite

Zusammenfassung: Das Kernstück deutscher Schulinspektionen ist die evaluation von Unterrichtsqualität mithilfe fragebogengestützter expertenratings. Die für die leistungsmessung zugrunde gelegten Qualitätsmerkmale rekurrieren zumeist auf bekannte Kriterienmatrizen effektiven Unterrichts und orientieren sich entsprechend am Prozess-Produkt-Paradigma der Schuleffektivitätsforschung. Bislang fehlen jedoch noch sowohl komplexe Unterrichtsqualitätsmodelle als auch maßstäbe für die leistungsbeurteilung und Standards für die leistungsbewertung, die es Schulverantwortlichen und Bildungsadministration erlauben, kriteriale Fragestellungen auf Basis der evaluationsergebnisse zu beantworten und die Schul-und Unterrichtsentwicklung anhand transparenter Kriterien wissensbasiert zu steuern. im vorliegenden Beitrag wird die idee aufgegriffen, auf grundlage vergleichender empirischer Forschung ein gestuftes modell von Unterrichtsqualität für die einordnung von evaluationsergebnissen zu erstellen. Das modell wird mithilfe der probabilistischen Testtheorie auf Basis von Daten einer normierungsstichprobe (n = 2240) der Schulinspektion Hamburg generiert. Die Befunde zeigen, dass es mit Daten aus Schulinspektionsverfahren grundsätzlich möglich ist, ein modell zu erstellen, das die Überführung quantitativer messwerte in qualitative Aussagen zur Qualität von Unterricht ermöglicht, das sich zwischen Schulformen invariant verhält und sowohl in den randbereichen der Skala als auch in deren mittelbereich gut diskriminiert. Die Datenstruktur ist jedoch mehrdimensional angelegt, sodass bei einer eindimensionalen modellierung von Unterrichtsqualität ein informationsverlust und Ungenauigkeiten bei der Bestimmung empirischer Kennwerte zu erwarten sind. gleichwohl sind die beobachteten Verzerrungen gering und die einzelnen Subdimensionen von Unterrichtsqualität korrelieren teilweise hoch miteinander, sodass davon auszugehen ist, dass ein eindimensionales Stufenmodell zur Beschreibung von Unterrichtsqualität eine sinnvolle, empirisch haltbare Approximation des mehrdimensionalen modells darstellt. ein solch abgestuftes modell wird abschließend mithilfe eines Proficiency Scaling aus den Daten heraus entwickelt und inhaltlich vorgestellt. Abstract:The core element in German school inspections is the evaluation of the quality of classroom teaching using questionnaire-supported expert ratings. The criteria for performance measurement are in most cases based upon research on effective teaching and are, therefore, Z erziehungswiss (2010) 13:121-148 122 m. Pietsch oriented on the process-product paradigm of school effectiveness research. Complex models for describing the quality of classroom teaching are missing as well as benchmarks for the assessment of and standards for the appraisal of performance, which allow leading personnel in school and education administration to answer questions based on the results of evaluations and to facilitate evidence-based governance and teaching development. This article suggests the development of a multi-level model f...

show abstract

“…Their approach did yield a description of "the behavior that the numerical value represents," in that it could be used to predict an examinee's likely success in spelling words not on the test. Other approaches to deriving substantive domain orderings from empirical item orderings are described by Fischer (1973), Embretson (1993), Tatsuoka (1990), Zwick, Senturk, Wang, & Loomis (2001), and others.…”

mentioning

confidence: 99%

Validating Standards-Based Test Score Interpretations

Haertel¹,

Lorie²

2004

Measurement: Interdisciplinary Research & Perspective

View full text Add to dashboard Cite

Standards-based score reports interpret test performance with reference to cut scores defining categories like "below basic" or "proficient" or "master." This paper first develops a conceptual framework for validity arguments supporting such interpretations, then presents three applications. Two of these are serve to introduce new standard-setting methods. Standards-based reporting has come into common use for educational assessments in the United States. Cut scores define performance levels for individual students, and the proportions of students at or above successive levels are reported for schools, states, and the entire nation. The broad appeal of such reporting is not surprising. Labels like "Basic," "Proficient," or "Advanced" seem to convey whether students are making satisfactory progress, whether schools are doing well enough, and in what subject areas students are most in need of improvement. In this standards-based era, it no longer seems sufficient to know whether annual scores are up or down; reporting in terms of quantified goals is called for to say how much better would be good enough. In the name of "ending social promotion," score thresholds are set to indicate which children need remediation and which are ready for the next grade level. And, at the individual student level, dozens of states have enacted requirements for high school exit examinations. Many of these standards have serious consequences; the potential for mischief is great if they are set capriciously. Because these cut scores define the decision rules according to which test scores are interpreted and used, the Standards for Educational and Psychological Testing state that "the validity of test interpretations may hinge on the cut scores" (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999, p. 53). The conceptual framework lays out the logic of validity arguments in support of standards-based score interpretations, focusing on requirements that the performance standard (i.e., the characterization of examinees who surpass the cut score) be defensible both as a description and as a normative judgmentUnfortunately, standards-based score reporting and student certification may be yet another case of political expectations outpacing measurement realities. It is arguable whether many such score interpretations are technically defensible. And while enthusiasm continues unabated, current standard-setting methods have been strenuously criticized (e.g., Berk, 1995;Glass, 1978;Jaeger, Mullis, Bourque, & Shakrani, 1996; National Academy of Education Panel on the Evaluation of the NAEP Trial State Assessment, 1993;Pellegrino, Jones, & Mitchell, 1999). This paper is divided into two major sections, the first presenting a conceptual framework and the second applying that framework to three standard-setting problems. After introducing the idea of validity argument, the first section takes up the related requirements for (1) a performance standard that desc...

show abstract

An Investigation of Alternative Methods for Item Mapping in the National Assessment of Educational Progress

Abstract: What is item mapping and how does it aid test score interpretation? Which item mapping technique produces the most consistent results and most closely matches expert opinion?

Cited by 39 publications

References 10 publications

Gender differences in performance in mathematics at the end of lower secondary school in Italy

Gender differences in performance in mathematics at the end of lower secondary school in Italy

Evaluation von Unterrichtsstandards

Validating Standards-Based Test Score Interpretations

Contact Info

Product

Resources

About