Quantifying Collaboration with a Co-Creative Drawing Agent

Abstract

This article describes a new technique for quantifying creative collaboration and applies it to the user study evaluation of a co-creative drawing agent. We present a cognitive framework called creative sense-making that provides a new method to visualize and quantify the interaction dynamics of creative collaboration, for example, the rhythm of interaction, style of turn taking, and the manner in which participants are mutually making sense of a situation. The creative sense-making framework includes a qualitative coding technique, interaction coding software, an analysis method, and the cognitive theory behind these applications. This framework and analysis method are applied to empirical studies of the Drawing Apprentice collaborative sketching system to compare human collaboration with a co-creative AI agent vs. a Wizard of Oz setup. The analysis demonstrates how the proposed technique can be used to analyze interaction data using continuous functions (e.g., integrations and moving averages) to measure and evaluate how collaborations unfold through time.

Complex Problem Solving through Human-AI Collaboration: Literature Review on Research Contexts

“…Reflecting on the characteristics of collaboration described earlier, an evolving process and reciprocity are essential. All these research context require a certain degree of sense-making, i.e., trying to understand the situation as well as actions and intentions of the other(s) and adjusting accordingly (e.g., match the intentioned drawing style [29]). They are also inherently iterative and interdependent, requiring a back-and-forth between agents.…”

“…Creative content creation Sketching [18,29], composing [32,33], writing [34,35] PO: Novelty, integrity, interestingness & balance [18,32], scariness [34] PB: Social & collaboration dynamics, flow [29,32,33] AP: Match writing style, novelty, creativity [35] I-TC: AI vs. WoZ [29] I-TC: AI vs. no-AI [18,33] I-TC: AI vs. human-AI vs. human [34] M-TP: Communication: inner state [32], suggestion style [35] Positive effect: Visualization of machine confidence on flow & composition [32] Positive effect: AI on novelty [18] and social dynamics [33] Positive effect: Hybrid approach on scariness [34] Writer prefer more fine-grained control over outputs and editability [35] Inconclusive: Group comparison; focus on framework [29] Turn-based cooperative games Card game (Hanabi), puzzle, word guessing [19][20][21] PO: Score, win/loose, efficiency [20,21] AP: Helpfulness, intelligence, sociability, humanness, likability, creativity, trustworthiness [19][20][21] I-TC: AI vs. presumed-human [20] M-TP: Communication: implicature & explanations [19,21] Positive effect: Implicature on score and perceived humanness [21] Positive effect: Explanation style on helpfulness, trustworthiness and overall experience [19] Negative attitude towards AI despite equal performance [20]…”

SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition

“…In the aspect of computer-aided sketch creation, Lee et al present ShadowDraw [30], a sketch drawing interface which guides users as they are drawing by ofering suggestions for object contours. Drawing Apprentice [9] is a sketching system that applies creative sense-making cognitive framework to explore human collaboration with a co-creative AI agent. Similarly, iSketch&Fill [17] constructs a drawing system by making shape recommendation and image generation through deep learning-based methods.…”