Assessing Knowledge and Application of the Design Process

Abstract

Authentic, client-based projects form the foundation of a one-semester freshman design course at Rice University. The course is an elective course available for all freshman students in the School of Engineering. First-year students learn the engineering design process and use it to solve meaningful problems drawn from local hospitals, industry, local community partners, Rice University, and international partners.The objectives for the course are to (a) have students learn and practice the engineering design process early in their engineering education, and (b) increase undergraduate retention in engineering at Rice University by 10 percentage points. These two objectives align with important themes and goals published elsewhere. For this first-year design course, three specific learning outcomes have been defined: students design a product that meets user-defined needs and realistic constraints; students communicate effectively through written reports and oral/visual presentations; and students work effectively on multidisciplinary teams.Steps in the design process form the core of the course lecture material. Most class periods are split between an interactive lecture about a step in the design process and team meetings to complete the design process, including prototype construction.Assessment of students' knowledge of the design process was measured by asking students to critique the strengths and weaknesses of a Gantt chart. The Gantt chart laid out a 14-week design process in which a team was to develop and build an examination bed for a clinic built by Engineers Without Borders in Nicaragua. Using this recently refined assessment tool, students' knowledge and application of the design process was evaluated at the beginning and end of the semester. Written responses were coded by three trained raters on eight aspects: 1) needs assessment/establishing design criteria; 2) design context review; 3) idea generation; 4) analysis and decision-making; 5) building and testing; 6) overall layout of a design process and iteration; 7) time allotments; and 8) documentation.Data presented in the paper includes 72 student responses from fall 2012 and spring 2013. Analysis shows statistical significance between pre-and post-test results for seven of the eight topics evaluated. The refined assessment method described here is an improvement over a previously deployed assessment tool.which is important for first-year engineering students. 12 As discussed below, we have modified Bailey's method to include a specific design challenge prompt.

Assessing Design Ability through a Quantitative Analysis of the Design Process

“…Studies that evaluate design ability through an understanding of the design process utilize either self-assessment [24,42] or qualitative measures, through an analysis of written texts or verbal forecasting of a design scenario [43][44][45][46][47]. Two studies utilize the critique of a Gantt chart to measure an understanding of the design process [48,49]. In Sims-Knight et al participants were asked to consider the amount of time spent in each step of the design process after completing a design activity [50].…”

Studying Changes using Concept Maps in First-Year Students' Understanding of the Engineering Design Process

“…Four main themes were identified as a framework for thematic analysis and they are described below. The first theme was identified because of the demand of social interactions of engineers with customers in the new era of customerization [1] and the recent emphasis of the entrepreneurial mindset across the nation [20][21][22] and details of the other themes were based on the criteria for assessing the design process knowledge described by Bailey and Szabo [14] and Saterbak and Volz [12], as well as many other studies about engineering design [23][24][25][26]. To ensure inter-rater reliability, two raters independently coded the data, labeling evidence and specific details of each theme in the data, and compared and reached consensus for any discrepancies.…”

“…[8][9][10][11]. Saterbak and Volts [12] and Zhu and Mertz [13] used students' critique of a Gantt chart that lays out a flawed proposed 14-week design process, an assessment tool developed by Bailey et al [14] to assess students design process knowledge in their introduction to engineering courses. Each of these approaches has its strengths and weaknesses [14].…”