Methods for eliciting and aggregating expert judgment are necessary when decision-relevant data are scarce. Such methods have been used for aggregating the judgments of a large, heterogeneous group of forecasters, as well as the multiple judgments produced from an individual forecaster. This paper addresses how multiple related individual forecasts can be used to improve aggregation of probabilities for a binary event across a set of forecasters. We extend previous efforts that use probabilistic incoherence of an individual forecaster's subjective probability judgments to weight and aggregate the judgments of multiple forecasters for the goal of increasing the accuracy of forecasts. With data from two studies, we describe an approach for eliciting extra probability judgments to (i) adjust the judgments of each individual forecaster, and (ii) assign weights to the judgments to aggregate over the entire set of forecasters. We show improvement of up to 30% over the established benchmark of a simple equal-weighted averaging of forecasts. We also describe how this method can be used to remedy the “fifty–fifty blip” that occurs when forecasters use the probability value of 0.5 to represent epistemic uncertainty.

With coordinated military and civil operations in developing countries, the integration of local stakeholder values with the goals of security and nation building is crucial. Such integration encourages innovative and effective courses of action and prioritization of resources. Although tools of multi-criteria decision analysis are well suited for resource prioritization, designing practical stakeholder value judgment elicitation in developing countries is a challenge because of cultural, organizational, and other barriers. Specifically, the weight of individual decision criteria can increase or decrease when diverse scenarios of emergent conditions are introduced or advocated by various stakeholders. This article develops methodology to identify the most important emergent conditions for infrastructure planning among a set of scenarios involving military and civil stakeholders. Across the infrastructure development alternatives, we identify scenarios that represent opportunities and those that represent threats. We adapt the framework of the swing weighting method for recalibration of a baseline value function with a variety of assumptions of scenarios, where each scenario is composed of one or more emergent conditions. The approach reduces the typical demand on stakeholders for elicitation of preference weights, as the entire value function is not entirely reconstructed per scenario. The approach and methodology were tested in a strategy workshop with more than 50 international participants and presented to ministry officials in a developing country. The testing integrated political, economic, environmental, and technology emergent conditions for prioritizing among infrastructure projects.

A recent paper in this journal described the identification and integration of sources of risk in a systems engineering process model [Lambert, Jennings, and Joshi, Syst Eng 9(3) (2006), 187–198]. The earlier effort falls short in addressing sources of deep, nonprobabilistic uncertainty that should enter to strategic systems design and reengineering. Our new paper incorporates the earlier effort to a framework for evaluating which are the deep uncertainties that most influence a priority‐setting among investments in large‐scale systems with multiple stakeholders, and therefore warrant more investigation. The framework addresses that deep uncertainties are continuously discovered and reflective of diverse and unique stakeholder experiences, knowledge bases, and advocacy positions. Deep uncertainties are epistemic viewpoints across which the strategic priorities for investments will differ. The framework modifies existing tools of scenario analysis and multicriteria analysis to process and filter the deep uncertainties. The framework is demonstrated in an application to reengineering of an energy system for a defense installation where frequent outages are disruptive to scientific and other missions. The sources of deep uncertainty in the demonstration include regulatory, economic, environment, cyber‐threat, and others. The investments include innovative microturbine and microgrid technologies. An example of a result is that international economic disruption is relatively more influential than cyber‐threats to strategic priority‐setting for investing in a microgrid at the particular installation. ©2012 Wiley Periodicals, Inc. Syst Eng 15

Military and industrial facilities need secure and reliable power generation. Grid outages can result in cascading infrastructure failures as well as security breaches and should be avoided. Adding redundancy and increasing reliability can require additional environmental, financial, logistical, and other considerations and resources. Uncertain scenarios consisting of emergent environmental conditions, regulatory changes, growth of regional energy demands, and other concerns result in further complications. Decisions on selecting energy alternatives are made on an ad hoc basis. The present work integrates scenario analysis and multiple criteria decision analysis (MCDA) to identify combinations of impactful emergent conditions and to perform a preliminary benefits analysis of energy and environmental security investments for industrial and military installations. Application of a traditional MCDA approach would require significant stakeholder elicitations under multiple uncertain scenarios. The approach proposed in this study develops and iteratively adjusts a scoring function for investment alternatives to find the scenarios with the most significant impacts on installation security. A robust prioritization of investment alternatives can be achieved by integrating stakeholder preferences and focusing modeling and decision-analytical tools on a few key emergent conditions and scenarios. The approach is described and demonstrated for a campus of several dozen interconnected industrial buildings within a major installation.