Preliminary Evaluation of the Psychometric Quality of HEIghtenTM Quantitative Literacy

Abstract

Quantitative literacy has been identified as an important student learning outcome (SLO) by both the higher education and workforce communities. This paper aims to provide preliminary evidence of the psychometric quality of the pilot forms for HEIghten quantitative literacy, a next-generation SLO assessment for students in higher education. We evaluated the psychometric quality of the test items (e.g., item analyses), individual-and group-level reliability, the relationship with student performance and related variables (e.g., grade point average) as well as student perceptions, and differences across college-related and demographic subgroups. Our study used data from a pilot test administered to over 1,500 students at 23 higher education institutions in the United States. Results showed that (a) overall, items were functioning well, but a small portion of items should be dropped due to unsatisfactory performance; (b) correlations across sub-areas of the assessment were very high indicating that the assessment may be unidimensional; (c) reliability estimates similar to existing SLO assessments were found at both individual and group levels; (d) assessment scores correlated positively with high school and college GPA, number of math college courses, self-rated quantitative literacy skills, and college admissions scores; (e) students had positive perceptions about the assessment; and (f) performance differences were found across institution type, college majors, gender, racial/ethnic groups, and language groups, but not across credit-hour categories. Implications for operational test development and understanding of quantitative literacy performance are discussed. Zhen Wang is a Senior Psychometrician at the Educational Testing Service (ETS). She holds a Ph.D. in Educational Measurement and Research Methodology from the University of British Columbia. Currently, she is the statistical coordinator for ETS' HEIghten® Outcomes Assessment Suite, Major Field Test, Success Navigator, Proficiency Profile, and iSkills projects. She also works on the research projects related to automated scoring and higher education. Her research interests include item response theory, structural equation models, hierarchical linear models, rater models, and equating and scaling techniques.

Concluding Thoughts

“…The historical framework of a progression of map types from reference maps to thematic maps is consistent with such a linguistic underpinning. After all, certainly a language of reference maps was essential to such activities as division of territory and navigation (Rankin 2016 for this and much more). Thematic maps are more concerned with conveying a message with location specificity as background (thus the very label, "reference"); widespread use of thematic maps is a relatively recent phenomenon.…”

“…Thematic maps are more concerned with conveying a message with location specificity as background (thus the very label, "reference"); widespread use of thematic maps is a relatively recent phenomenon. Finally, map literacy has traditionally been the purview of a relatively specialized group of cartographers and geographers with perhaps more traditional notions of what constitutes map literacy and indeed a map (Rankin 2016).…”

“…Our fourth major thought came relatively late in the project and was largely triggered by our awareness and appreciation of a new book, After the Map (Rankin 2016). The author, who is a historian, cartographer, and master illustrator, has produced a work which, we believe, is destined to sit alongside the classic Flattening the Earth (Snyder 1993) on bookshelves of must-read books on cartography and representational maps (Rankin's term for reference maps).…”

“…For us, the takeaway is that the distinction between reference (L > T) maps and thematic (T > L) maps is still relevant not only because it is the traditional one of textbook writers (e.g., Dent et al 2009); it is relevant because it is fundamental. The distinction will survive even as what maps are used for moves around our triangle with changing technologies and even as the very concept of territory evolves as Rankin (2016) argues. In other words, the L > T maps are representational, and T > L maps are presentational, meaning their purpose is to present thematic information on what amounts to be a representational background.…”

“…Meanwhile, on the ground, that characterization leads to word problems being used by mathematicians to bring context they understand to the math concepts they are teaching in math class (Karaali and Vacher 2020). It also leads to word problems used to assess skills one would like to think students would learn in their math classes (e.g., Gaze et al 2014;Roohr et al 2017).…”