A Decision Tree Based Methodology for Evaluating Creativity in Engineering Design

Abstract

Multiple metrics have been proposed to measure the creativity of products, yet there is still a need for effective, reliable methods to assess the originality of new product designs. In the present article we introduce a method to assess the originality of concepts that are produced during idea generation activities within engineering design. This originality scoring method uses a decision tree that is centered around distinguishing design innovations at the system level. We describe the history and the development of our originality scoring method, and provide evidence of its reliability and validity. A full protocol is provided, including training procedures for coders and multiple examples of coded concepts that received different originality scores. We summarize data from over 500 concepts for garbage collection systems that were scored by Kershaw et al. (2015). We then show how the originality scoring method can be applied to a different design problem. Our originality scoring method, the Decision Tree for Originality Assessment in Design (DTOAD), has been a useful tool to identify differences in originality between various cohorts of Mechanical Engineering students. The DTOAD reveals cross-sectional differences in creativity between beginning and advanced students, and shows longitudinal growth in creativity from the beginning to the end of the undergraduate career, thus showing how creativity can be influenced by the curriculum. The DTOAD can be applied to concepts produced using different ideation procedures, including concepts produced both with and without a baseline example product, and concepts produced when individuals are primed to think of different users for their designs. Finally, we show how our the DTOAD compares to other measurements of creativity, such as novelty, fixation, and remoteness of association.

Engineering Design Concept Generation: The Effect of Concept Combination and Classification

“…Then we calculate the quantity of concepts produced in each round per group, per student. Kershaw et al [23] defined product concept radically different if it received a score of 7.5 or 10. We also compare the quantity of radically different concepts from Round 1 to 2.…”

“…It uses a 5 point ordinal scale to represent lowest to highest originality score ranging from '0' to '10' with 2.5 increment between each level as shown in Figure 1. Kershaw et al [23] achieved the inter rater reliability of r= 0.70 and validated DTOAD metric. DTOAD compares concepts with existing products in the market rather than comparing within sample alone.…”

“…[22] Brown [18] agreed to this limitation along with few more. Kershaw et al [23] created the Decision Tree for Originality Assessment in Design (DTOAD). DTOAD ( Figure 1) allows system level originality assessment rather than feature level by focusing on assessing if the innovation is beyond the industry norm and then how integral the innovation is to the concept.…”

“…The concepts were assessed for originality using the Decision Tree for Originality Assessment in Design (DTOAD) metric. [23] Before assessment, we trained raters similar to the procedure used by Kershaw et al [23]. We used two raters for this experiment.…”

“…It is understandable that not many students can produce very original concepts at first. We further investigated if, interventions contributed to reducing the quantity of non-original concepts scoring '0' or '2.5' and increase the quantity of radically different [23] product concepts scoring 7.5 and above. Table 5 shows the summary of concepts produced in each round with rating from 0 to 7.5, recall there was no concept with 10 score.…”