Structuring, Aggregating, and Evaluating Crowdsourced Design Critique

Abstract

Feedback is an important component of the design process, but gaining access to high-quality critique outside a classroom or firm is challenging. We present CrowdCrit, a webbased system that allows designers to receive design critiques from non-expert crowd workers. We evaluated CrowdCrit in three studies focusing on the designer's experience and benefits of the critiques. In the first study, we compared crowd and expert critiques and found evidence that aggregated crowd critique approaches expert critique. In a second study, we found that designers who got crowd feedback perceived that it improved their design process. The third study showed that designers were enthusiastic about crowd critiques and used them to change their designs. We conclude with implications for the design of crowd feedback services.

Evaluating Information Visualization via the Interplay of Heuristic Evaluation and Question-Based Scoring

“…Recent work has shown how to better align novice critiques with that of experts; in particular, the use of rubrics has been shown in massive open online courses to bring student ratings within one grade of instructors [18] and novice guideline-based feedback in aggregate can be as acceptable to the receivers as expert feedback [30,21]. However, positively written and emotional critiques received higher average ratings than negative or neutral critiques [33].…”

Apparition

“…The crowd itself can also provide feedback. Prior work has explored how individuals can get feedback from crowd workers during the design process [22].…”

“…Crowdsourcing inherently provides access to individuals with diverse skills, which provides guidance to users during the design process [22] and improves specific skills like drawing. Limpaechter et al collected sketch data from a crowd of players of their DrawAFriend game and used the resulting model to correct user-drawn lines in real-time [19].…”

The Analysis and Presentation of Patents to Support Engineering Design

“…Crowd rely heavily on the idea tree for inspiration, and best ideas appear around ends of the idea tree. Luther et al [65] Help designers to receive design critiques from non-expert crowd workers.…”