Consistency and Predictive Nature of Vertically Moderated Standards for South Carolina's 1999 Palmetto Achievement Challenge Tests of Language Arts and Mathematics

Huynh, Huynh; Barton, Karen; Meyer, Joseph P.; Porchea, Sameano; Gallant, Dorinda J.

doi:10.1207/s15324818ame1801_7

Cited by 5 publications

(4 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Scores on the PACT can range from 536 to 664 and are categorized on four levels: Below Basic I and II (not prepared for the next grade), Basic (minimally prepared), Proficient (prepared), and Advanced (highly prepared). Huynh, Barton, Meyer, Porchea, and Gallant (2005) found that 98.5% of the sixth-grade students who scored Proficient or Advanced on the math portion of the PACT passed seventh grade the following year. Approximately, 11.5% of sixth-grade students who scored Below Basic I passed seventh grade, 34.3% of sixth-grade students scoring Below Basic II passed seventh grade, and 74.4% of sixth-grade students who scored Basic passed seventh grade.…”

Section: M-cbmsmentioning

confidence: 98%

Aggregated Randomly Generated Math Curriculum–Based Measurements for Middle School Students: Reliability, Predictive Validity, and Cut Score Precision

Strait

Smith

McQuillin

2018

Assessment for Effective Intervention

View full text Add to dashboard Cite

Interventioncentral.org receives more than a million visits a year (Strait et al., 2015). One of its most commonly used resources-receiving more than 100,000 visits-is the math worksheet generator. On their web page content area, Curriculum-Based Measurement (CBM) Warehouse, interventioncentral.org advertises the math worksheet generator as an online CBM application (Intervention Central, 2017; see http://www.interventioncentral.org/curriculum-basedmeasurement-reading-math-assesment-tests). In general, CBMs are brief and standardized skill-based measures that correspond to an entire curriculum (Deno, 2003; Foegen & Deno, 2001). Given their brevity, CBMs are used to measure students' response to interventions and identify students in need of intervention (Hosp, Hosp, & Howell, 2016; Shapiro, 2011). Recently, Strait and colleagues (2015) evaluated the reliability of randomly generated math-CBMs (M-CBMs) from interventioncentral.org's math worksheet generator. The M-CBMs corresponded with a remedial math intervention curriculum for sixth-grade students with skill deficits in addition, subtraction, multiplication, and division. Strait and colleagues reported that aggregating (averaging) two to three randomly generated M-CBMs created a single reliable (i.e., test-retest reliability) score greater than >.80, whereas averaging four resulted in reliability estimates exceeding .90 (Strait, 2008). Unfortunately, having to give multiple M-CBMs to obtain a single reliable score (i.e., >.80) is inefficient and decreases the measure's functionality as a progress-monitoring tool-challenging the categorization of these assessments as M-CBMs (Strait et al., 2015). However, in this article, given that interventioncentral.org advertises these measures as CBMs and they were designed as brief and standardized measures for a remedial math curriculum, we continue to refer to them as M-CBMs-albeit M-CBMs with questionable technical adequacy. As such, this study attempts to provide more information on the technical adequacy and utility of these M-CBMs to identify students in need of intervention. Currently, it is assumed based on the Spearman-Brown prophecy formula (Strait et al., 2015) that using aggregated M-CBMs would improve predictive validity by improving test-retest reliability, but this assumption is untested.

show abstract

Section: M-cbmsmentioning

confidence: 98%

Aggregated Randomly Generated Math Curriculum–Based Measurements for Middle School Students: Reliability, Predictive Validity, and Cut Score Precision

Strait

Smith

McQuillin

2018

Assessment for Effective Intervention

View full text Add to dashboard Cite

show abstract

“…Some early proponents of vertical articulation methods (e.g., Bejar et al, 2007;Huynh, Barton, Meyer, Porchea, & Gallant, 2005;Lewis & Haug, 2005) have recommended a holistic approach from the beginning. They argued that the following steps in test development should be considered from a holistic perspective, and when attention is paid to each step, the steps logically support articulation of cut scores.…”

Section: Articulation In the Context Of The Next-generation K-12 Assementioning

confidence: 99%

Vertical Articulation of Cut Scores Across the Grades: Current Practices and Methodological Implications in the Light of the Next Generation of K–12 Assessments

Kannan

2016

ETS Research Report Series

View full text Add to dashboard Cite

Federal accountability requirements after the No Child Left Behind (NCLB) Act of 2001 and the need to report progress for various disaggregated subgroups of students meant that the methods used to set and articulate performance standards across the grades must be revisited. Several solutions that involve either a priori deliberations or post‐hoc adjustments have been offered over the years. In this paper, I provide a methodological review of the alternative cut‐score articulation methods, including some novel solutions (e.g., using predictive methods) that have been proposed in the context of the next‐generation K–12 assessments. In systematically evaluating these methods, I focus on the psychometric challenges they might present and the practical feasibility of their operational implementation. In addition, results from a survey of several state departments of education help to provide information on the prevalence of these methods across the states. Overall, this review shows that none of the alternative methods is completely free of limitation; yet, each method provides solutions that are appropriate for addressing certain methodological and practical requirements. Therefore, in the context of the next‐generation assessments and the need to identify students who are on track to being college and career ready, practitioners are advised to consider a combination of methods and cautioned against overreliance on any single method.

show abstract

“…Moreover, in the context of the next‐generation assessments, several researchers (e.g., Bejar, Braun, & Tannenbaum, ; Huynh, Barton, Meyer, Porchea, & Gallant, ; Lewis & Haug, ) have recommended the use of holistic approaches that integrate educational policy, learning theory, and curriculum design in developing content and performance standards. Two such approaches, which involve the articulation of curriculum and test design, serve as promising solutions for the articulation of performance standards: (a) curricular (or content standard) articulation and (b) use of vertical scales in developing tests across the grades.…”

Section: Introductionmentioning

confidence: 99%

Articulation of Cut Scores in the Context of the Next‐Generation Assessments

Kannan

Sgammato

2017

ETS Research Report Series

View full text Add to dashboard Cite

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. Peer review notwithstanding, the positions expressed in the ETS Research Report series and other published accounts of ETS research are those of the authors and not necessarily those of the Officers and Trustees of Educational Testing Service.The Daniel Eignor Editorship is named in honor of Dr. Daniel R. Eignor, who from 2001 until 2011 served the Research and Development division as Editor for the ETS Research Report series. The Eignor Editorship has been created to recognize the pivotal leadership role that Dr. Eignor played in the research publication process at ETS. ETS Research Report Series ISSN 2330-8516 R E S E A R C H R E P O R T Articulation of Cut Scores in the Context of the Next-Generation Assessments Priya Kannan & Adrienne SgammatoEducational Testing Service, Princeton, NJ Logistic regression (LR)-based methods have become increasingly popular for predicting and articulating cut scores. However, the precision of predictive relationships is largely dependent on the underlying correlations between the predictor and the criterion. In two simulation studies, we evaluated the impact of varying the underlying grade-level correlations on the resultant bias in cut scores articulated using the LR method. In Study 1, we compared different articulation methods (LR and equipercentile smoothing), and in Study 2, we evaluated different criteria for linking (e.g., adjacent grade or end of course). The collective results indicate that as correlations became smaller, cut scores articulated using LR-based predictions became increasingly biased when compared to a true value obtained under perfect correlation. The predicted values are significantly biased for lower achievement levels, irrespective of the linking criteria used. Results from these studies suggest that the LR method must be used with caution, particularly when articulating cut scores for multiple achievement levels.

show abstract

Consistency and Predictive Nature of Vertically Moderated Standards for South Carolina's 1999 Palmetto Achievement Challenge Tests of Language Arts and Mathematics

Cited by 5 publications

References 3 publications

Aggregated Randomly Generated Math Curriculum–Based Measurements for Middle School Students: Reliability, Predictive Validity, and Cut Score Precision

Aggregated Randomly Generated Math Curriculum–Based Measurements for Middle School Students: Reliability, Predictive Validity, and Cut Score Precision

Vertical Articulation of Cut Scores Across the Grades: Current Practices and Methodological Implications in the Light of the Next Generation of K–12 Assessments

Articulation of Cut Scores in the Context of the Next‐Generation Assessments

Contact Info

Product

Resources

About