A Generalizability Theory Approach to Standard Error Estimates for Bookmark Standard Settings

Lee, Guemin; Lewis, Daniel M.

doi:10.1177/0013164407312603

Cited by 17 publications

(16 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Several studies (e.g., Brennan & Lockwood, 1980;Clauser, Swanson, & Harik, 2002;Hurtz & Hertz, 1999;Lee & Lewis, 2008;Maurer, Alexander, Callahan, Bailey, & Dambrot, 1991;Norcini, Lipner, Langdon, & Strecker, 1987;Norcini, Shea, & Grosso, 1991;Norcini, Shea, & Ping, 1988) have applied G-theory to investigate the reliability of cut-score estimates. However, most of these studies have not investigated the generalizability of cut scores over random samples of items.…”

Section: G-theory Applied To Angoff Standard Settingmentioning

confidence: 99%

Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items Within a Generalizability Theory Framework

Kannan

Sgammato

Tannenbaum

et al. 2015

Applied Measurement in Education

View full text Add to dashboard Cite

The Angoff method requires experts to view every item on the test and make a probability judgment. This can be time consuming when there are large numbers of items on the test. In this study, a G-theory framework was used to determine if a subset of items can be used to make generalizable cut-score recommendations. Angoff ratings (i.e., probability judgments) from previously conducted standard setting studies were used first in a re-sampling study, followed by D-studies. For the re-sampling study, proportionally stratified subsets of items were extracted under various sampling and test-length conditions. The mean cut score, variance components, expected standard error (SE) around the mean cut score, and root-mean-squared deviation (RMSD) across 1,000 replications were estimated at each study condition. The SE and the RMSD decreased as the number of items increased, but this reduction tapered off after approximately 45 items. Subsequently, D-studies were performed on the same datasets. The expected SE was computed at various test lengths. Results from both studies are consistent with previous research indicating that between 40-50 items are sufficient to make generalizable cut score recommendations.

show abstract

Section: G-theory Applied To Angoff Standard Settingmentioning

confidence: 99%

Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items Within a Generalizability Theory Framework

Kannan

Sgammato

Tannenbaum

et al. 2015

Applied Measurement in Education

View full text Add to dashboard Cite

show abstract

“…The intent of measurement under subject-centered student testing is to differentiate persons (test takers) while generalizing over items whereas the intent Correspondence between generalizability study designs for stimulus-centered standard setting and subjectcentered student testing is shown in Table 4. An example of a stimulus-centered application of generalizability theory is provided by Lee and Lewis (2001). This transposition is of great practical importance for how tests and standard-setting studies are developed.…”

Section: Variability In Resultsmentioning

confidence: 99%

“…Correspondence between generalizability study designs for stimulus‐centered standard setting and subject‐centered student testing is shown in Table 4. An example of a stimulus‐centered application of generalizability theory is provided by Lee and Lewis (2001).…”

Section: A Fresh Look At Standard‐setting Criticismsmentioning

confidence: 99%

See 1 more Smart Citation

Standard‐Setting Methods as Measurement Processes

Nichols

Twing

Mueller

et al. 2010

Educational Measurement

View full text Add to dashboard Cite

Some writers in the measurement literature have been skeptical of the meaningfulness of achievement standards and described the standard‐setting process as blatantly arbitrary. We argue that standard setting is more appropriately conceived of as a measurement process similar to student assessment. The construct being measured is the panelists' representation of student performance at the threshold of an achievement level. In the first section of this paper, we argue that standard setting is an example of stimulus‐centered measurement. In the second section, we elaborate on this idea by comparing some popular standard‐setting methods to the stimulus‐centered scaling methods known as psychophysical scaling. In the third section, we use the lens of standard setting as a measurement process to take a fresh look at the two criticisms of standard setting: the role of judgment and the variability of results. In the fourth section, we offer a vision of standard‐setting research and practice as grounded in the theory and practice of educational measurement.

show abstract

“…: design, the two variance components structures were based on a persons within groups within forms study in a standard setting context (Lee & Lewis, 2008). Lastly, for the ( )…”

Section: Variance Component Structures and Sample Size Patternsmentioning

confidence: 99%

Estimating standard errors of estimated variance components in generalizability theory using bootstrap procedures

Moore¹

View full text Add to dashboard Cite

This study investigated the extent to which rules proposed by Tong and Brennan (2007) for estimating standard errors of estimated variance components held up across a variety of G theory designs, variance component structures, sample size patterns, and data types. Simulated data was generated for all combinations of conditions, and point estimates, standard error estimates, and coverage for three types of confidence intervals were calculated for each estimated variance component and relative and absolute error variance across a variety of bootstrap procedures for each combination of conditions. It was found that, with some exceptions, Tong and Brennan's (2007) rules produced adequate standard error estimates for normal and polytomous data, while some of the results differed for dichotomous data. Additionally, some refinements to the rules were suggested with respect to nested designs. This study provides support for the use of bootstrap procedures for estimating standard errors of estimated variance components when data are not normally distributed.

show abstract

A Generalizability Theory Approach to Standard Error Estimates for Bookmark Standard Settings

Cited by 17 publications

References 11 publications

Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items Within a Generalizability Theory Framework

Evaluating the Consistency of Angoff-Based Cut Scores Using Subsets of Items Within a Generalizability Theory Framework

Standard‐Setting Methods as Measurement Processes

Estimating standard errors of estimated variance components in generalizability theory using bootstrap procedures

Contact Info

Product

Resources

About