Online administration of a reasoning inventory in development

Abstract

We are developing the Physics Inventory of Quantitative Literacy (PIQL), a new research based assessment (RBA) focused on quantitative reasoning-rather than conceptual understanding-in physics contexts. We rapidly moved administration of the PIQL online in Spring 2020 due to the COVID-19 pandemic. We present our experiences with online, unproctored administration of an RBA in development to students enrolled in a large-enrollment, calculus-based, introductory physics course. We describe our attempts to adhere to best practices on a limited time frame, and present a preliminary analysis of the results, comparing results from the online administration to earlier results from in-person, proctored administration. We include discussion of online administration of multiple-choice/multiple-response (MCMR) items, which we use on the instrument as a way to probe multiple facets of student reasoning. Our initial comparison indicates little difference between online and paper administrations of the PIQL, except for performance MCMR items

Effect of administration method on student performance on multiple-choice/multiple-response items

“…The PIQL was originally developed as an in-person, proctored assessment, but shifted to online administration in early 2020 due to the COVID-19 pandemic. Though the shift was not planned, online administration of research-based assessments offers many advantages (namely, a smaller investment of time and resources by those administering the test) [5]. We therefore began to collect evidence of the PIQL's validity as an online assessment, with an eye toward dissemination of the PIQL as an online assessment instrument for widespread use.…”

“…As such, our preliminary analyses of online responses to MCMR items focused on the effectiveness of the measures taken to inform students about the nature of those items. The preliminary analysis found that the measures were effective: not only were students choosing more than one response on the MCMR items administered online, they were choosing more than one response at a higher rate than students seeing the same items in-person [5]. Because we grouped the MCMR items together at the end of the PIQL on the online version (a change from the in-person version), we were not sure whether this was due to the change in administration method from inperson to online, or due the change in question order.…”

Toward understanding and characterizing expert physics covariational reasoning

“…Connecting a quantity's physical meaning to its mathematical representation is an important part of developing and making sense of mathematical models, and has been shown to be challenging for many students in introductory physics contexts and beyond [20,[22][23][24][25]. Research has demonstrated student difficulties with making sense of the physical meaning of vectors, noting that many students can make sense of the symbols but struggle with translating between vector and graphical notation and extrapolating the information to a vector field [26].…”

“…The blended space may also include utterances that are inseparable by the researchers. Increasingly, physics education researchers are using the term more broadly to refer to the blended nature of mathematical structure and physical meaning [24,25,44,45].…”