Advancing Assessment of Quantitative and Scientific Reasoning

Abstract

Advancing Assessment of Quantitative and Scientific Reasoning is a four-year NSF Project (DUE-0618599) in part designed to evaluate the generalizability of quantitative (QR) and scientific reasoning (SR) assessment instruments created at James Madison University to four other four-year institutions with very distinct missions and student demographics. This article describes the methods, results, and findings we obtained in our studies. More specifically, we describe how to conduct content-alignment exercises in which faculty members map each item from a prospective test to the student learning objectives taught at the institution. Our results indicated that 92-100% of the QR and SR items were successfully mapped to each of the partner institutions’ learning objectives. We also guided the partner institutions on assessing the balance of test items across the intended student learning objectives to assure greater content validity and coverage. The reliability (internal consistency) results from the partner institutions for the learning objectives and major subtests are strikingly similar across very different student populations. We interpret lower reliabilities from one institution to be the result of test administration and student motivation factors, the latter being a serious threat to the health and vigor of any assessment program. Validity study results at the partner institutions add to the evidence of construct validity of the QR and SR instruments. While our studies focus on QR and SR instruments, the methods will apply to other instruments and other institutions as they attempt to answer important questions about student learning outcomes

Developing a Test of Scientific Literacy Skills (TOSLS): Measuring Undergraduates’ Evaluation of Scientific Information and Arguments

“…, 2004; Miller, 2007). General scientific reasoning instruments were developed specifically to test cognitive skills related to critical thinking and reasoning (Lawson, 1978; Facione, 1991; Sundre, 2003, 2008; Sundre et al. , 2008; Quitadamo et al.…”

Assessing College Students' Quantitative and Scientific Reasoning: The James Madison University Story

“…Included in the Assessment Day tests is the Natural World-9 (NW-9), a 66-item test developed by JMU faculty and the Center for Assessment and Research Studies to assess Cluster 3's QR and SR learning objectives 3 (Table 1; Sundre and Thelk 2010). Specifically, the NW-9 is composed of two tests: QR-9 and SR-9.…”

“…Thus, the sum of all items is representative of scientific reasoning (QR+SR), as defined; this total is referred to as the NW-9. The QR-9, SR-9 and NW-9 4 have been extensively studied and are all currently in their ninth versions (see Sundre and Thelk 2010;Hurney et al 2011). For clarity, we will refer to these tests as the QR and the QR+SR in this paper.…”

“…We have studied QR-SR for over 20 years. Throughout this time we have refined our assessment methodologies (Sundre and Thelk 2010) and accumulated a growing body of evidence about student learning outcomes. This evidence also highlights issues with QR-SR that warrant further investigation in the context of higher education.…”

Comparative Study of the Numeracy Education and Writing Across the Curriculum Movements: Ideas for Future Growth

“…For example, in this journal, Taylor (2009) provides an overview of some of the welldeveloped measures for quantitative literacy assessment, and Sundre and Thelk (2010) discuss the technicalities and importance in aligning QR assessment with learning goals. These are just two of the many examples in QR assessment.…”