A Simulation Study of Computer-Supported Inferential Analysis Under Data Overload

Abstract

A simulation study of inferential analysis under data overload was conducted with professional intelligence analysts. Using a process tracing methodology, patterns in information sampling and sources of inaccurate statements were identified when analysts were asked to analyze something outside their base of expertise, were tasked with a tight deadline, and had a large data set. The main contribution from this study is a better understanding of potential vulnerabilities in inferential analysis in challenging situations. These vulnerabilities are informative because they point to a set of design criteria that human-centered solutions to data overload should meet in order to be useful. These evaluation criteria are interesting, in part, because they are so difficult to address. They are not amenable to simple, straightforward adjustments or feature additions to current tools. Meeting these design criteria will require innovative design concepts.

Human Factors and Ergonomics in Automation Design

“…These issues can be summarized as the need to broaden searches to enhance opportunity to discover highly relevant information, promote recognition of unexpected information to avoid premature fixation on a particular viewpoint or hypothesis, and manage data uncertainty to avoid misinterpretation of inaccurate or obsolete data (Woods et al, 1999). These represent important challenges that may require innovative design concepts and significant departures from current tools (Patterson, 1999). Just as swarm automation might help explore physical spaces, it might also help explore information spaces.…”

“…Current approaches to searching large complex data sources, such as the Internet, are limited. People are likely to miss important documents, disregard data that represent a significant departure from initial assumptions, misinterpret data that conflict with an emerging understanding, and disregard more recent data that could revise interpretation (Patterson, 1999). These issues can be summarized as the need to broaden searches to enhance opportunity to discover highly relevant information, promote recognition of unexpected information to avoid premature fixation on a particular viewpoint or hypothesis, and manage data uncertainty to avoid misinterpretation of inaccurate or obsolete data (Woods et al, 1999).…”

Emerging challenges in cognitive ergonomics: Managing swarms of self-organizing agent-based automation

“…Current approaches to searching large complex data sources, such as the Internet, are ine ective. People are likely to miss important documents, disregard data that is a signi®cant departure from initial assumptions , misinterpret data that corroborates or con¯icts with an emerging understanding , and disregard more recent data that could revise interpretation (Patterson 1999). These issues can be summarized as the need to broaden searches to enhance opportunity to discover highly relevant information, promote recognition of unexpected information to avoid premature ®xation on a particular viewpoint or hypothesis, and manage data uncertainty to avoid misinterpretation of inaccurate or obsolete data (Woods et al 1999).…”

“…These issues can be summarized as the need to broaden searches to enhance opportunity to discover highly relevant information, promote recognition of unexpected information to avoid premature ®xation on a particular viewpoint or hypothesis, and manage data uncertainty to avoid misinterpretation of inaccurate or obsolete data (Woods et al 1999). These represent important challenges that may require innovative design concepts and signi®cant departures from current tools (Patterson 1999).…”

Cognitive Engineering Challenges of Managing Swarms of Self-Organizing Agent-Based Automation

“…Current approaches to searching large complex data sources, such as the Internet, are ineffective. People are likely to miss important documents, disregard data that is a significant departure from initial assumptions, misinterpret data that corroborates or conflicts with an emerging understanding, and disregard more recent data that could revise interpretation (Patterson, 1999). These issues can be summarized as the need to broaden searches to enhance opporhmiry to discover highly relevant information, promote recognition of unexpected information to avoid premature fixation on a particular viewpoint or hypothesis, and manage data uncertainfy to avoid misinterpretation of inaccurate or obsolete data (Woods, Patterson, Roth, & Christoffersen, 1999).…”