How Effective Are Decision Analyses? Assessing Decision Process and Group Alignment Effects

Abstract

In this paper, we analyze approaches to assess the effectiveness of decision analyses. We develop an effectiveness framework, categorized in metrics to assess the quality of the decision process (“process effectiveness”), the quality of the model results (“output effectiveness”), and the quality of the long-term consequences of the analysis (“outcome effectiveness”). With a focus on the first two dimensions, “process” and “output,” we introduce two new approaches to assess the effectiveness of decision analyses applied in organizations. First, a new process effectiveness approach serves to compare the effectiveness of decision analyses to existing decision processes. Second, we assess output effectiveness with a before/after preference measurement design, which aids to capture the alignment of group members quantitatively when using decision analysis. We used six case studies, implemented during an applied research project, to test the two measures.

Ranking the harm of non-medically used prescription opioids in the UK

“…decision conferencing.Research on decision conferences (Schilling et al, 2007, Chun, 1992, McCartt and Rohrbaugh, 1995, McCartt and Rohrbaugh, 1989 and group workshops (Franco and Montibeller, 2015, Phillips, 2011, Schilling et al, 2007 has identified four features that contribute to validity: competence of the experts (Shanteau, 1992), diversity of perspectives in the group (Shanteau, 2001), impartial facilitation by a facilitator who guides the process but does not contribute to content (Phillips, 2007), and development on-the-spot of a model that is based on sound theory (Regan-Cirincione, 1994), in this case, decision theory (Dodgson et al, 2000, Keeney andRaiffa, 1976).All of these factors were applied, as far as was practically possible, in designing and conducting the decision conference that was used for this study.…”

Asking Questions: A Sine Qua Non of Facilitation in Decision Support?

“…It is difficult, however, to clearly establish the link between context, process and outcomes of decision support (Schilling et al 2007). In natural settings, the context in which a facilitated modelling approach is used and how the approach is used in that context, is largely beyond the control of researchers (Finlay 1998;Rouwette et al 2009).…”

From the Editors—Group Decisions, Preference Elicitation, Experienced Utility, Survival Probabilities, and Portfolio Value of Information

“…The result of this paper will encourage a prescriptive decision analysis approach to group decisions resulting in more thoughtful decisions. Prior Decision Analysis papers on group decisions include Bordley (2009), Dias and Sarabando (2012), Ewing and Baker (2009), Rios andRios (2009), andSchilling et al (2007).…”