Exploring student understanding of the engineering design process using distractor analysis

Abstract

Typical approaches to assessing students' understanding of the engineering design process (EDP) include performance assessments that are time-consuming to score. It is also possible to use multiple-choice (MC) items to assess the EDP, but researchers and practitioners often view the diagnostic value of this assessment format as limited. However, through the use of distractor analysis, it is possible to glean additional insights into student conceptualizations of complex concepts. Using an EDP assessment based on MC items, this study illustrates the value of distractor analysis for exploring students' understanding of the EDP. Specifically, we analyzed 128 seventh grade students' responses to 20 MC items using a distractor analysis technique based on Rasch measurement theory. Our results indicated that students with different levels of achievement have substantively different conceptualizations of the EDP, where there were different probabilities for selecting various answer choices among students with low, medium, and high relative achievement. We also observed statistically significant differences (p < 0.05) in student achievement on several items when we analyzed the data separately by teacher. For these items, we observed different patterns of answer choice probabilities in each classroom. Qualitative results from student interviews corroborated many of the results from the distractor analyses. Together, these results indicated that distractor analysis is a useful approach to explore students' conceptualization of the EDP, and that this technique provides insight into differences in student achievement across subgroups. We discuss the results in terms of their implications for research and practice.

Effects of infusing the engineering design process into STEM project-based learning to develop preservice technology teachers’ engineering design thinking

“…Interested in cultivating engineering design thinking in university and secondary school students (Wind, Alemdar, Lingle, Moore, & Asilkalkan, 2019), many researchers have examined the differences in design thinking processes between expert and novice engineers when faced with an engineering problem. For example, Atman et al (2005), Atman et al (2007), and Lammi and Becker (2013) used engineering problems such as the design of a playground for a fictitious neighborhood and the construction of a ping-pong-ball launcher to investigate the engineering design thinking of engineering experts and students.…”

Design and Design Thinking in STEM Education

“…Given the fact that previous studies mainly focused on professional designers and engineers' cognition, studies on students' design thinking and its development are still limited. Existing studies have illustrated that this is a rich and fruitful area for scholarly discussion and research (e.g., Kavousi et al 2019;Strimel et al 2019;Wind et al 2019). Systematic studies on students' design thinking and its development, especially in and through STEM education, would help provide important foundations for developing sound educational programs and instruction.…”

Rethinking Educational Assessment from the Perspective of Design Thinking

“…On 'Force and motion', for example, learning outcomes are interpreted based on a metacommunicative account of various levels of understanding (Alonzo & Steedle, 2009, p. 39;cf. di Uccio et al, 2020;Wilson, 2009;Wind et al, 2019):…”

“…Augmented with supplemental information on item content and a construct map illustrating the relevant learning progressions (Alonzo & Steedle, 2009;Black et al, 2011;Ketterlin-Geller et al, 2019;Wilson, 2005Wilson, , 2009 di Uccio et al, 2020), the Figure 2 (Black & Wiliam, 1998. Items incorporating partial credit assignments, such as 107 and 108 in Figure 2, may offer added value by alerting the student and teacher to the presence of misconceptions (Andrich & Styles, 2011;Masters, 1982;Wind et al, 2019). Each student's pattern of responses may be unique without compromising fit of the data to the measurement model or the explanatory power of the predictive construct theory.…”