Almost an Expert

Abstract

Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2×2 factorial design. Student designers received feedback on a visual design from both experts and novices, who produced feedback using either an expert rubric or no rubric. We found that rubrics helped novice workers provide feedback that was rated nearly as valuable as expert feedback. A follow-up analysis on writing style showed that student designers found feedback most helpful when it was emotionally positive and specific, and that a rubric increased the occurrence of these characteristics in feedback. The analysis also found that expertise correlated with longer critiques, but not the other favorable characteristics. An informal evaluation indicates that experts may instead have produced value by providing clearer justifications.

Evaluating Information Visualization via the Interplay of Heuristic Evaluation and Question-Based Scoring

“…However, positively written and emotional critiques received higher average ratings than negative or neutral critiques [33]. To counter this, Robb et al [28] found that designers seeking feedback tended to shut out negative textual feedback, but responded well to visual feedback that showed critics did not understand their designs.…”

Supporting answerers with feedback in social Q&A

“…Answerers did not always recognize the utility of raters in providing an outsider assessment. Criteria-based, aggregated crowd feedback approached expert feedback in both our study and prior work [12,20]. Designs should establish the legitimacy of raters.…”

“…Luther et al [12], in a study of the crowdworker-sourced critique system CrowdCrit, found that aggregated crowd critique approached expert critique, and designers who received crowd feedback perceived that it improved their design process. In a study of design feedback by Yuan et al [20], crowdworkers providing feedback without criteria tended to focus on surface-level features, while with criteria, their reviews were as valuable as expert reviews. In our study, we hope to effectively deploy crowd feedback in a setting where peer feedback is rare, through the use of aggregated crowdworker feedback with criteria targeted toward promoting effective help-giving.…”

“…In a mixed methods study, we explored the use of crowdworkers and tested the effectiveness of a criteria-based feedback intervention on Brainly answerers. Crowdworkers are a promising source of scalable feedback for MOOCs and other learning settings, and have shown the capability to give expert-level feedback when given criteria [20].…”

Creative design software: Challenges and opportunities to the graphic designer in Zimbabwe

“…As such, this breaks down the geographical and cost barriers between designers and clients. Traditionally, designers used to receive critiques and feedback through studio critique sessions, where they would present their work to peers and mentors who would in turn provide comments and suggestions (Yuan et al, 2016). Crowdsourcing has proved to be a readily available method for effective feedback which has led some researchers to explore crowdsourcing as a potential solution to design problems (Luther et al, 2015).…”