Impact of Scenario Selection on Robustness

Abstract

Multiple plausible future scenarios are being used increasingly in preference to a single deterministic or probabilistic prediction of the future in the long-term planning of water resources systems. These scenarios enable the determination of the robustness of a system-the consideration of performance across a range of plausible futures-and allow an assessment of which possible future system configurations result in a greater level of robustness. There are many approaches to selecting scenarios, and previous studies have observed that the choice of scenarios might affect the estimated robustness of the system. However, these observations have been anecdotal and qualitative. This paper develops a systematic, quantitative methodology for exploring the influence of scenario selection on the robustness and the ranking of decision alternatives. The methodology is illustrated on the Lake Problem. The quantitative results obtained confirm the qualitative observations of previous works, showing that the selection of scenarios is important, as it has a large influence on the robustness value calculated for each decision alternative. However, we show that it has a relatively small influence on how those decision alternatives are ranked. This implies that despite the difference in robustness values, similar decision outcomes will be reached in this case study, regardless of the basis on which the scenarios are obtained. It is also revealed that the impact of the scenarios on the robustness values is due to complex interactions with the system model and robustness metrics.

Robust decision outcomes with induced correlations in climatic and economic parameters

“…Furthermore, some conclusions from scenario generation procedures appear consistent across the different case studies evaluated; for instance, the three studies cited above all find that uniformly distributed scenarios result in different robustness scores and vulnerability characterizations than centrally distributed scenarios. However, they differ in terms of the impact they observe on robustness rankings, with Quinn et al (2020) showing a high influence on robustness rankings that is not apparent in McPhail et al (2020) or Reis and Shortridge (2020). This suggests that the degree to which scenario generation methods influence DMDU outcomes is likely to vary on a case-by-case basis, and points toward the need for multiple comparisons that can highlight common patterns and themes.…”

“…For instance, robustness measures are generally categorized as either satisficing-or regret-based (Herman et al 2015). Within each of these broad categories, there are multiple mathematical definitions for robustness that quantify how a given alternative performs across the ensemble of scenarios evaluated (McPhail et al 2018(McPhail et al , 2020. These mathematical quantifications allow for comparison of different alternatives, but multiple studies have demonstrated that the choice of robustness metric can influence robustness scores (i.e., how robust an individual or group of alternatives appear to be) and rankings (which alternatives appear most robust) (Drouet et al 2015;Giuliani and Castelletti 2016;Kwakkel et al 2016a, b;McPhail et al 2018).…”

“…Similar issues were explored by Quinn et al (2020), who compared vulnerability analysis results from scenarios based on historic streamflow observations, paleoclimate data, and climate change projections. They show that the method used to generate scenarios can impact both the uncertain parameters that drive vulnerability as well as robustness rankings (Quinn et al 2020). McPhail et al (2020 proposed a systematic methodology for quantifying the influence that scenario selection procedures have on robustness scores and rankings.…”

“…McPhail et al (2020 proposed a systematic methodology for quantifying the influence that scenario selection procedures have on robustness scores and rankings. They demonstrate this procedure across an experimental setup that compares the number and distribution of scenarios used to simulate a conceptual model of lake quality, finding that robustness scores are relatively sensitive to scenario distribution, while robustness rankings were relatively insensitive (McPhail et al 2020).…”

“…These studies have demonstrated the degree to which scenario generation procedures can impact DMDU outcomes, but several gaps limit the extent to which this understanding can be translated into improvements in DMDU practice. For instance, uncertain parameters were correlated in some of the experimental designs evaluated by Quinn et al (2020) andMcPhail et al (2020), but these designs also differed in terms of the sample size, shape, and range of these parameters' marginal distributions. This makes it impossible to discern the influence of input parameter correlation relative to the other factors that differed across the experimental setups evaluated in each study.…”