Designing upper-division thermal physics assessment items informed by faculty perspectives of key content coverage

Abstract

Though several conceptual inventories have been developed for thermal physics, none target upperdivision material and all focus specifically on thermodynamics without including statistical mechanics content. In this paper, we outline the development process of an upper-division thermal physics assessment that captures both thermodynamics and statistical mechanics content. To ensure that the assessment can be broadly usable, as a first step in this process, we administered a survey to physics faculty to determine the scope and content variability of thermal physics courses across institutions. We received over 70 responses from 63 unique institutions, approximately half of which are minority-serving institutions and women's colleges. Our findings support the claim that there is significant variation in the content covered at different institutions, but also some general agreement on a number of core content areas. We identified 10 key topics which were listed by the majority (95%) of survey respondents to focus on for the assessment development process. We then wrote assessment objectives that encompass core content goals within these 10 topics, which guided the writing of free-response assessment items that were piloted with students. Using student responses to the free-response items, we developed multiple-response versions, which includes both multiple-choice and coupled, multiple-response items. In this paper, we present details of the faculty survey, including methods of developing and distributing the survey to solicit a broad range of perspectives. Additionally, we present results of the survey, including core content covered by faculty in upper-division thermal physics courses, and discuss how these results were used to guide development of assessment objectives and assessment items. We include the full development process of one assessment item as an example.

Methods for utilizing Item response theory with Coupled, Multiple-Response assessments

“…The development and validation of the U-STEP using Classical Test Theory are reported in Refs. [9,13,14]. An exploratory factor analysis of data from the U-STEP indicated that the U-STEP's factor structure is dominated by a single factor, and thus, the instrument appears statistically unidimensional.…”

Applying Voting Theory to Mastery Grading; A Study of Faculty Interpretation of Course-Level Categorical-Score Distributions

“…TaSPA was developed with specific attention to how faculty use assessment results to inform changes to their instruction based on theories of self-regulated learning and evidence-centered design (ECD) [2,[5][6][7]. TaSPA evaluates students with respect to their achievement of a learning goal categorically as: "Met"(M), "Partially Met"(P), or "Not Met"(N); referred to together as MPN categories.…”

Affordances of Articulating Assessment Objectives in Research-based Assessment Development

“…As described in Ref. [4], AOs are written to "collectively span the space of content areas identified as important based on" the domain analysis and to be "directly TABLE I. A subset of AOs from the U-STEP and SPRUCEwith example data that informed articulation of those AOs and example items targeting those AOs.…”

“…Since the early 1990's, research-based assessments have helped inform research in physics education [1], and, in recent years, both the pace of assessment development and use of assessments by instructors and researchers has grown [2][3][4][5][6]. Typically, these assessments are not intended to evaluate individual students for the purpose of assigning grades [7]; rather they aim to help course instructors and education researchers identify areas for, or evaluate the effectiveness of, reforms and interventions [8][9][10][11].…”

“…As examples of the critical role that AOs can play in developing assessment items, as well as in the design process more broadly, we discuss two recent research-based assessments that were developed using AOs. The two assessments discussed are a thermodynamics assessment, the Upper-level Statistical Mechanics and Thermodynamics Evaluation for Physics (U-STEP) [4] and the Survey of Physics Reasoning on Uncertainty Concepts in Experiments (SPRUCE), which is intended for use in lower-division physics labs.…”