Criterion‐Related Validity: Assessing the Value of Subscores

Abstract

Criterion-related profile analysis (CPA) Lately, there has been much interest in subscores, because they can potentially provide diagnostically useful information. A subscore refers to a score on one test in a test battery or one section of a single test. A score profile x p = (X 1 , X 2 , . . . , X V ) is the vector of V subscores for person p, where p = 1, . . . , P and v = 1, . . . , V, and the total for person p is a sum of the subscores X p = v X pv . Recent literature shows at least three lines of subscore research. First, researchers have proposed methods that yield subscores with improved psychometric properties, usually reliability (de la Torre

Can We Learn From Student Mistakes in a Formative, Reading Comprehension Assessment?

“…To address this question, Biancarosa et al. () employed a logistic regression analysis of incorrect answer profiles (Davison, Davenport, Chang, Vue, & Su, ). For purposes of this analysis, it was recognized that students could be scored as incorrect for one of three reasons: a paraphrase response, a lateral connection response, or not completing the item.…”

“…Having shown that incorrect answer scores can be reliable, we then turned to the question of whether those scores provide additional information not available from a simple NC score. To address this question, Biancarosa et al (2019) employed a logistic regression analysis of incorrect answer profiles (Davison, Davenport, Chang, Vue, & Su, 2015). For purposes of this analysis, it was recognized that students could be scored as incorrect for one of three reasons: a paraphrase response, a lateral connection response, or not completing the item.…”

aPRIDIT Unsupervised Classification with Asymmetric Valuation of Variable Discriminatory Worth

“…conjunction with other variables, such as demographics, if subsequent investigative analysis is desired. According to Davison, Davenport, Chang, Vue, and Su (2015), the usefulness of a scoring methodology (such as aPRIDIT provides for each individual) is increased if the scores can be used for predicting an external criterion. By contrast, Kohonen classification is graphical in nature (Brockett et al, 1998), and would be hard to incorporate into other numerical statistical analysis.…”

Subscore equating with the random groups design

“…Some studies have evaluated the added value of subscores based on reliability (Brennan, 2011(Brennan, , 2015Feinberg & Wainer, 2014;Haberman, 2008;Haberman et al, 2009;Sinharay, 2014). Other studies have focused on validity of subscores (Davison, Davenport, Chang, Vue, & Su, 2015;Haladyna & Kramer, 2004). Haberman (2008) proposed PRMSE, an index indicating the added value of subscores over the total scores.…”

“…The author applied the suggested method to data from several operational tests and found that the added value of subscores with respect to classification was very similar to their added value using Haberman's approach, except at extreme cut scores. Davison et al (2015) assessed the value of subscores in predicting an external criteria. They conducted a criterion-related profile analysis (CPA) to determine whether subscores account for more criterion variance than does the total score.…”