User Testing with Assessors to Develop Universal Rubric Rows for Assessing Engineering Design

Abstract

This paper describes the process of testing and refining modular rubric rows developed for the assessment of engineering design activities. This is one component of a larger project to develop universal analytic rubrics for valid and reliable competency assessment across different academic disciplines and years of study. The project is being undertaken by researchers based in the Faculty of Applied Science and Engineering at the University of Toronto.From January 2014 to June 2015, we defined and validated indicators (criteria) for engineering design, communication, and teamwork learning outcomes, then created descriptors for each indicator to discriminate between four levels of performance: Fails, Below, Meets, and Exceeds graduate expectations. From this rubric bank, applicable rows can be selected and compiled to produce a rubric tailored to a particular assessment activity. Here we discuss these rubrics within the larger context of learning outcomes assessment tools for engineering design.We tested draft rubrics in focus group sessions with assessors (teaching assistants and course instructors who assess student work in engineering design). We followed the testing with structured discussions to elicit feedback on the quality and usability of these rubrics, and to investigate how the assessors interpreted the language used in the indicators and descriptors. We asked participants to identify indicators they believed were irrelevant, redundant, or missing from the rubric. We also asked them to identify and discuss indicators and descriptors that were confusing. Finally, we asked them what changes they would recommend and what training materials they would find useful when using rubrics of this design.We analysed the consistency of assessor ratings to identify problematic rows and used qualitative feedback from follow-up discussions to better understand the issues that assessors had with the rubrics. Three of the six engineering design rubric items we tested showed evidence of rater inconsistency, uncertainty, and indecision. While some rubric rows received similar criticism from most participants, we also identified differences in assessors' rubric design preferences and in how they apply rubrics to evaluate student work. It also emerged that assessors have different conceptions of engineering design and the design process, and are confused when presented with unfamiliar terminology. Based on our interpretations, we identified changes we should make to improve the rubric structure and content.Our aim is to inform other educators who may be developing tools for the assessment of engineering design by sharing our methodology and discussing the feedback received. Engineering educators could adopt or adapt our user testing methodology to improve the usability of similar assessment tools. Our discoveries about rubric structure improvements could be explored further to define best practices in the design of universal rubrics. Our next steps include applying what we have learned to refine the rubrics and devel...

Engineering Instructor Use and Interpretation of Rubrics in a Large Scale Undergraduate Independent Research Course

“…An interview protocol was developed based on previous research investigating how assessors use and interpret universal rubrics [4,6]. The interview protocol first posed demographic and experience questions, including questions about the supervisors' experience using rubrics (in the thesis course or otherwise).…”

Update on the Development of Analytic Rubrics for Competency Assessment (Darca)

“…The purpose of this paper is to describe this process and to present the preliminary results of this activity (for a detailed analysis of the feedback received on the design rubric, see [1]). …”

STEAM student recruitment through a pre-college customer-centric design enrichment experience (WIP)

“…Previous efforts in literature have resulted in quantitative instruments for assessment of design classes useful in guiding future improvements, but these suffer from low scalability due to requiring direct observations or interviews [2]. This contrasts with other literature focused primarily on assessing skills developed by participants [3], [4], or developing rubrics and methods for qualitative assessment of student work product generated in design focused activities [5] and classes [6].…”