No abstract

No abstract

This study is an evaluation of the psychometric issues associated with estimating objective level scores, often referred to as ‘‘subscores.’’ The article begins by introducing the concepts of reliability and validity for subscores from statewide achievement tests. These issues are discussed with reference to popular scaling techniques, classical test theory, and item response theory. Methods for increasing the reliability of subscore estimates that have been suggested in literature are then reviewed. Based on this review, an empirical study comparing some of the more promising procedures was conducted. Test score data from a large statewide testing program were analyzed in this study. The comparison of subscore augmentation approaches found that generally all methods were very successful in dramatically increasing the reliability of subscore estimates. However, this increase was accompanied by near-perfect correlations among the subscore estimates. This finding called into question the validity of the resultant subscores, and therefore the usefulness of the subscore augmentation process. Implications for practice are discussed.

In many behavioral research areas, multivariate generalizability theory (mG theory) has been typically used to investigate the reliability of certain multidimensional assessments. However, traditional mG-theory estimation-namely, using frequentist approaches-has limits, leading researchers to fail to take full advantage of the information that mG theory can offer regarding the reliability of measurements. Alternatively, Bayesian methods provide more information than frequentist approaches can offer. This article presents instructional guidelines on how to implement mG-theory analyses in a Bayesian framework; in particular, BUGS code is presented to fit commonly seen designs from mG theory, including single-facet designs, two-facet crossed designs, and two-facet nested designs. In addition to concrete examples that are closely related to the selected designs and the corresponding BUGS code, a simulated dataset is provided to demonstrate the utility and advantages of the Bayesian approach. This article is intended to serve as a tutorial reference for applied researchers and methodologists conducting mG-theory studies.

No abstract

The equating performance of two internal anchor test structures-miditests and minitests-is studied for four IRT equating methods using simulated data. Originally proposed by Sinharay and Holland, miditests are anchors that have the same mean difficulty as the overall test but less variance in item difficulties. Four popular IRT equating methods were tested, and both the means and SDs of the true ability of the group to be equated were varied. We evaluate equating accuracy marginally and conditional on true ability. Our results suggest miditests perform about as well as traditional minitests for most conditions. Findings are discussed in terms of comparability to the typical minitest design and the trade-off between accuracy and flexibility in test construction.

In this note, we demonstrate an interesting use of the posterior distributions (and corresponding posterior samples of proficiency) that are yielded by fitting a fully Bayesian test scoring model to a complex assessment. Specifically, we examine the efficacy of the test in combination with the specific passing score that was chosen through expert judgment, or, in general, any external a priori criterion. In addition, we study the robustness of the test's efficacy with respect to choice of the passing score.

When missing values are present in item response data, there are a number of ways one might impute a correct or incorrect response to a multiple-choice item. There are significantly fewer methods for imputing the actual response option an examinee may have provided if he or she had not omitted the item either purposely or accidentally. This article applies the multiple-choice model, a multiparameter logistic model that allows for in-depth distractor analyses, to impute response options for missing data in multiple-choice items. Following a general introduction of the issues involved with missing data, the article describes the details of the multiple-choice model and demonstrates its use for multiple imputation of missing item responses. A simple simulation example is provided to demonstrate the accuracy of the imputation method by comparing true item difficulties ( p values) and item–total correlations ( r values) to those estimated after imputation. Missing data are simulated according to three different types of missing mechanisms: missing completely at random, missing at random, and missing not at random.