The CACHE Study: Group Effects in Computer-supported Collaborative Analysis

Abstract

The present experiment investigates effects of group composition in computersupported collaborative intelligence analysis. Human cognition, though highly adaptive, is also quite limited, leading to systematic errors and limitations in performance -that is, biases. We experimentally investigated the impact of group composition on an individual's bias, by composing groups that differ in whether their members initial beliefs are diverse (heterogeneous group) or similar (homogeneous group). We study three-member, distributed, computer-supported teams in heterogeneous, homogeneous, and solo (or nominal) groups. We measured bias in final judgment, and also in the selection and evaluation of the evidence that contributed to the final beliefs. The distributed teams collaborated via CACHE-A, a web-based software environment that supports a collaborative version of Analysis of Competing Hypotheses (or ACH, a method used by intelligence analysts). Individuals in Heterogeneous Groups showed no net process cost, relative to noninteracting individuals. Both heterogeneous and solo (noninteracting) groups debiased strongly, given a stream of balanced evidence. In contrast, individuals in Homogenous Groups did worst, accentuating their initial bias rather than debiasing. We offer suggestions about how CACHE-A supports collaborative analysis, and how experimental investigation in this research area can contribute to design of CSCW systems.

Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study

“…potentially plausible "narratives", in our earlier terminology). ACH--based tools clearly scaffold knowledge--building discourse in a disciplined way, for instance, CACHE (Convertino et al 2008;Shrager, et al 2010), provides a collaborative ACH space for the exploration of hypotheses in open science, uses notification spreading through provenance chains in order to simplify revision updating, marking questionable results, and informing scientists when their hypothesis or claims potentially need to be revised. Interestingly, just as intelligence analysis tools such as ACH provide support for detailed analysis of competing options that might be expressed in a collaborative discourse platform such as Cohere, we note that Smallman (2008, p.334) notes the need within the intelligence analysis community for better support in argument analysis and visualization, of the sort provided by Cohere, or Rationale (van Gelder 2002).…”

Effects of implicit sharing in collaborative analysis

“…Such sharing can improve both process and performance in collaborative sensemaking and analysis tasks, measured by accuracy of task outcomes [19,39] and recall of decisions [13]. Sharing can also help analysts flexibly organize data to discover insights and form schemas [39,40] or concept maps [15], identify and link evidence from different sources [4,6], identify entities in the data [9], and encounter otherwise hidden and overlooked connections between pieces of information [1,9,11,25], leading to better decisions [19]. The success of shared workspaces in team performance has been attributed to promoting exchange of information and data with others [19], as a result improved common ground [39] and awareness of the status of the analysis task and others' activities in the task [13,34].…”

“…This is a general problem for evaluating collaborative analysis tools: the high stakes and established processes of real investigations makes it hard to deploy new tools in the field. Thus, tool evaluations tend to involve short-term tasks completed by university students [1,11,12,13,19,23,39]. These controlled contexts and tasks do provide compensating advantages, such as the ability to systematically vary tools in ways that allow researchers to carefully examine specific factors that together build a body of knowledge to inform real world tools design.…”

“…One option would be to combine implicitly shared, unstructured notes with explicit support for formal hypotheses as provided by CACHE [11], Jigsaw [35], and Sandbox [40]. As with the NLP techniques described above, having semantically different representations for notes and inferences might help analysts separate these activities, reducing the tendency to fixate on hypotheses early and increasing the chance that analysts develop competing hypotheses, which is important to final outcomes [11,20]. Implicitly shared notes could also be used as resources for creating, supporting, or refuting these hypotheses, with the system suggesting connections between existing notes and newly proposed hypotheses.…”

“…Most previous work evaluated collaborative analysis tools with a handful of participants without comparing tools to alternatives [3,9,34,40], by running a solo study [23], by studying explicit knowledge sharing [11,13], or have not evaluated the tools with human participants at all [30,32,35]. In this paper we present a methodical study that demonstrated the benefit of a specific design feature, implicit knowledge sharing, on the process and outcomes of a collaborative analysis task.…”