Improving Intelligence Analysis With Decision Science

Abstract

Intelligence analysis plays a vital role in policy decision making. Key functions of intelligence analysis include accurately forecasting significant events, appropriately characterizing the uncertainties inherent in such forecasts, and effectively communicating those probabilistic forecasts to stakeholders. We review decision research on probabilistic forecasting and uncertainty communication, drawing attention to findings that could be used to reform intelligence processes and contribute to more effective intelligence oversight. We recommend that the intelligence community (IC) regularly and quantitatively monitor its forecasting accuracy to better understand how well it is achieving its functions. We also recommend that the IC use decision science to improve these functions (namely, forecasting and communication of intelligence estimates made under conditions of uncertainty). In the case of forecasting, decision research offers suggestions for improvement that involve interventions on data (e.g., transforming forecasts to debias them) and behavior (e.g., via selection, training, and effective team structuring). In the case of uncertainty communication, the literature suggests that current intelligence procedures, which emphasize the use of verbal probabilities, are ineffective. The IC should, therefore, leverage research that points to ways in which verbal probability use may be improved as well as exploring the use of numerical probabilities wherever feasible.

Complex Problem Solving: What It Is and What It Is Not

“…There are interesting options to be explored (again): think-aloud protocols , which have been discredited for many years (Nisbett and Wilson, 1977) and yet are a valuable source for theory testing (Ericsson and Simon, 1983); introspection (Jäkel and Schreiber, 2013), which seems to be banned from psychological methods but nevertheless offers insights into thought processes; the use of life-streaming (Wendt, 2017), a medium in which streamers generate a video stream of think-aloud data in computer-gaming; political decision-making (Dhami et al, 2015) that demonstrates error-proneness in groups; historical case studies (Dörner and Güss, 2011) that give insights into the thinking styles of political leaders; the use of the critical incident technique (Reuschenbach, 2008) to construct complex scenarios; and simulations with different degrees of fidelity (Gray, 2002). …”

A Survey of Intelligence Analysts' Perceptions of Analytic Tools

“…Future research could also explore how individual features of this multi-faceted concept determine analysts' perceptions of available tools. As with other recent efforts to move towards an evidence-based approach to intelligence analysis (see [24]), there ought to be an evidence-based approach to analytic technology. The findings of such research can be used to inform the design and development of tools that meet analysts' requirements.…”

Critical Review of Analytic Techniques

“…First, although the underlying assumption is that analysts using these techniques will perform better than those that do not, the effectiveness of these techniques has rarely been empirically tested (for exceptions of tests in the intelligence domain, see [7,8,9]). Therefore, there is an urgent need to empirically test the effectiveness of these analytic techniques [10]. The present paper highlights a criterion on which a technique could be tested (i.e., by its primary function), and suggests the sort of analytic task that could be used to test a technique i.e., a task at the stage of the workflow where that technique is best applied.…”