Enhancing Scoring Reliability In Mid Program Assessment Of Design

Davis, Denny; Trevisan, Michael S.; McKenzie, Larry; Beyerlein, Steven

doi:10.18260/1-2--9218

Cited by 6 publications

(3 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Results of this survey compare favorably with previous work using other TIDEE assessment tools, which includes a Design Team Readiness Assessment 3,4 that measures team design skills of student teams as they engage in a short design project, team interviews , a verbal protocol analysis 6 , and continuous classroom improvement assessment activities 2 . Deployment of the Design Team Growth Survey provides a relatively quick and reliable feedback mechanism with a goal of increasing student learning through the improvement of the curriculum, development of better classroom management techniques and creation of more effective methods and pedagogies.…”

Section: Introductionmentioning

confidence: 75%

Measuring Added Value Using A Team Design Skills Growth Survey

Gentili¹,

Lyons²

2005 Annual Conference Proceedings

Self Cite

View full text Add to dashboard Cite

Assessing student learning in design courses is essential for giving them feedback on their integration of a wide range of knowledge and skills. This paper describes the Team Design Skills Growth Survey, which is easy to administer and interpret in discerning design capabilities of students in introductory engineering design classes. This tool measures student perceptions about their professional growth and correlates these with perceived course emphasis on learning outcomes for design skills, teamwork skills, and communication skills endorsed by the Washington Council for Engineering and Related Technology Education (WCERTE).Several versions of the Team Design Skills Growth Survey have been used over the last ten years in sections of an introductory engineering design course at Tacoma Community College. Results generated from the survey are consistent with pre-and post-testing, verbal protocol analysis, team interviews, and a variety of reflective writing assignments. Results include an analysis of the difference between the means for class-averaged growth and class-averaged emphasis in each of the WCERTE outcome areas.For students, the greatest impact of using this tool is increasing their understanding of their learning with respect to the intended learning outcomes. As such, the Team Design Skills Growth Survey can support efforts to increase students' self-confidence in engineering, leading to better retention of engineering students. For faculty, the greatest impact of using this tool is measuring the value-added of different activities in introductory engineering design classes, which can be used to improve course design, provide insight to class management issues, and determine the appropriateness of performance indicators. This paper also addresses best practices for implementing and expanding the Team Design Skills Growth Survey and extending it to other disciplines.

show abstract

Section: Introductionmentioning

confidence: 75%

Measuring Added Value Using A Team Design Skills Growth Survey

Gentili¹,

Lyons²

2005 Annual Conference Proceedings

Self Cite

View full text Add to dashboard Cite

show abstract

“…For our purposes, transition activity measured in MacShapa is used as a surrogate measure for design iteration 26 . The scoring of the TIDEE Mid-Program Assessment self-reports for each team was completed by individuals trained in the scoring process 15,37 . Documents were scored "blind": scorers did not have access to the videotape sessions.…”

Section: Methodsmentioning

confidence: 99%

Comparing Design Team Self Reports With Actual Performance: Cross Validating Assessment Instruments

Adams¹,

Punnakanta²,

Atman³

et al.

2002 Annual Conference Proceedings

View full text Add to dashboard Cite

Assessing student learning of the engineering design process is challenging. Students' ability to answer test questions about the design process or record their design activities may differ significantly from their actual performance in solving "messy" open-ended problems. In the Pacific Northwest, multi-university participants in a National Science Foundation supported project (Transferable Integrated Design Engineering Education, TIDEE) have implemented and disseminated a Mid-Program Assessment instrument for assessing engineering student design competency. One part of the instrument requires student teams to document (e.g., self-report) their design decisions and processes while engaged in a design task. These written self-reports are scored using a rubric that has demonstrated a high inter-rater reliability. We are interested in comparing the scores derived from these selfreports with measures of actual design performance. Our research method for analyzing design performance is verbal protocol analysis. In this study, eighteen teams of students (2-6 students per team) from four different institutions were videotaped as they completed the TIDEE Mid-Program Assessment. In this paper we provide 1) a description of the assessment instrument, 2) our research methods for assessing the validity of the instrument, 3) examples of comparing self-reports to performance, and 4) a summary of our findings. We conclude with a discussion of the strengths and weaknesses of this study, as well as implications for teaching and assessing engineering student design competency.

show abstract

“…Detailed reliability and validity studies of the Design Team Readiness Assessment have been discussed elsewhere 7,8 . Three raters participated in a multi-step procedure that included initial scoring of student work, reconciliation of differences among raters, revision of scoring criteria, and the development of decision rules to deal with student work that appears difficult to score within the scoring criteria.…”

Section: Design Team Readiness Assessmentmentioning

confidence: 99%

Best Practices In Implementing An Engineering Mid Program Assessment

Rutar¹,

Beyerlein²,

Thompson³

et al.

2002 Annual Conference Proceedings

Self Cite

View full text Add to dashboard Cite

This paper outlines practices that ensure quality in administering, interpreting, reporting, and maintaining the 'Design Team Readiness Assessment' developed by the Transferable Integrated Design Engineering Education (TIDEE) consortium in the Pacific Northwest. A copy of the instrument can be downloaded from www.cea.wsu.edu/tidee. The instrument assesses design process, teamwork, and design communication skills in three different contexts. Previous work has demonstrated how to achieve high inter-rater reliability through explicit scoring criteria and decision rules. For this reason, the 'Design Team Readiness Assessment' can be used to evaluate the preparation of beginning and mid-level engineering and engineering technology students across institutions and degree programs. Faculty who have implemented the instrument have found it to be a valuable classroom tool, promoting self-awareness of life-long learning skills in a variety of course settings and supporting action research on lower-division design experiences. Their discoveries are summarized here using a framework for assessment literacy that is widely used in the K-12 education community.

show abstract

Enhancing Scoring Reliability In Mid Program Assessment Of Design

Cited by 6 publications

References 7 publications

Measuring Added Value Using A Team Design Skills Growth Survey

Measuring Added Value Using A Team Design Skills Growth Survey

Comparing Design Team Self Reports With Actual Performance: Cross Validating Assessment Instruments

Best Practices In Implementing An Engineering Mid Program Assessment

Contact Info

Product

Resources

About