This paper is part of the Focused Collection on Quantitative Methods in PER: A Critical Examination.] A central aim of physics education research is to understand the processes of learning and use that understanding to inform instruction. To this end, researchers often conduct studies to measure the effect of classroom interventions on student outcomes. Many of these intervention studies have provided an empirical foundation of reformed teaching techniques, such as active engagement. However, many times there is not sufficient evidence to conclude that the intervention had the intended effect, and these null results often end up in the proverbial file drawer. In this paper, we argue that null results can make significant contributions to physics education research, even if the results are not statistically significant. First, we review social science and biomedical research that documents widespread publication bias against null results, exploring why it occurs and how it can hurt the field. We then present three cases from physics education research to highlight how studies that yield null results can contribute to our understanding of teaching and learning. Finally, we distill from these studies some general principles for learning from null results, proposing that we should evaluate them not on whether they reject the null hypothesis but according to their potential for generating new understanding.
Abstract. Current step-by-step, problem-solving frameworks have been shown to improve aspects of students' problem solving. However, conceptual problem-solving shortcuts that break from these frameworks may demonstrate adaptive problem solving. Following an approach from Heckler [1], we show that prompting introductory physics students to complete a common initial step of these frameworks ("draw a free-body diagram") before solving force problems decreases the use of conceptual shortcuts and increases reproduction of standard procedures. Extending this paradigm, students were then asked to evaluate a sample informal solution for a force problem. When the problem text included a diagram prompt, students rated the informal solution less favorably, commenting that the solution was not formal enough. These results imply that the prompt to draw a diagram not only cues procedural problem-solving approaches, but also may push students away from informal approaches that demonstrate problem-solving expertise.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.