Comparing Design Team Self Reports With Actual Performance: Cross Validating Assessment Instruments

Abstract

Assessing student learning of the engineering design process is challenging. Students' ability to answer test questions about the design process or record their design activities may differ significantly from their actual performance in solving "messy" open-ended problems. In the Pacific Northwest, multi-university participants in a National Science Foundation supported project (Transferable Integrated Design Engineering Education, TIDEE) have implemented and disseminated a Mid-Program Assessment instrument for assessing engineering student design competency. One part of the instrument requires student teams to document (e.g., self-report) their design decisions and processes while engaged in a design task. These written self-reports are scored using a rubric that has demonstrated a high inter-rater reliability. We are interested in comparing the scores derived from these selfreports with measures of actual design performance. Our research method for analyzing design performance is verbal protocol analysis. In this study, eighteen teams of students (2-6 students per team) from four different institutions were videotaped as they completed the TIDEE Mid-Program Assessment. In this paper we provide 1) a description of the assessment instrument, 2) our research methods for assessing the validity of the instrument, 3) examples of comparing self-reports to performance, and 4) a summary of our findings. We conclude with a discussion of the strengths and weaknesses of this study, as well as implications for teaching and assessing engineering student design competency.

Assessing Knowledge and Application of the Design Process

“…22 Contextualizing the prompt is consistent with much of the literature on assessing design process knowledge. 13,14,16,17 Figure 1. Pre-and post-test assessment prompt used in fall 2012 and thereafter.…”

“…These methods include surveys, interviews, talk aloud protocols, concept maps, exams and written reports, as well as the evaluation of students' final design prototypes. [12][13][14][15][16][17] Each of these methods also has well documented limitations. Combining several methods to cross-validate results compensates for these limitations; however, triangulating and analyzing multiple sources of data require considerable time and resources.…”

Experiencing Capstone Design Problem Statements

“…While the simple answer is that all of this is required for project success, how much of this design method is a problem statement and how much is engineering specifications? For Adams, et al 18 , it is the problem definition that describes what the problem really is, what are the constraints, and what are the (performance) criteria. Woods 13 , proposes that the define "stage" includes the stated objective, context, constraints, criteria (inputs and outputs) with a focus on classification of given information and not on understanding what really is the problem.…”

Quickly Building Students’ Confidence in their Fabrication Abilities

“…There is a large and growing body of evidence that early project-based experiences (such as firstyear, cornerstone courses) are benefitting engineering students. They serve to enhance student interest in engineering, improve retention, and improve results in later courses [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18] ; that evidence is reviewed now. Assessment and outcomes research has been done more vigorously in recent years (see 19 for a comprehensive survey), and there have been many studies of design per se (e.g.…”