Exploring How Component Factors and Their Uncertainty Affect Judgements of Risk in Cyber-Security

Abstract

Subjective judgements from experts provide essential information when assessing and modelling threats in respect to cyber-physical systems. For example, the vulnerability of individual system components can be described using multiple factors, such as complexity, technological maturity, and the availability of tools to aid an attack. Such information is useful for determining attack risk, but much of it is challenging to acquire automatically and instead must be collected through expert assessments. However, most experts inherently carry some degree of uncertainty in their assessments. For example, it is impossible to be certain precisely how many tools are available to aid an attack. Traditional methods of capturing subjective judgements through choices such as high, medium or low do not enable experts to quantify their uncertainty. However, it is important to measure the range of uncertainty surrounding responses in order to appropriately inform system vulnerability analysis. We use a recently introduced interval-valued response-format to capture uncertainty in experts' judgements and employ inferential statistical approaches to analyse the data. We identify key attributes that contribute to hop vulnerability in cyber-systems and demonstrate the value of capturing the uncertainty around these attributes. We find that this uncertainty is not only predictive of uncertainty in the overall vulnerability of a given system component, but also significantly informs ratings of overall component vulnerability itself. We propose that these methods and associated insights can be employed in real world situations, including vulnerability assessments of cyber-physical systems, which are becoming increasingly complex and integrated into society, making them particularly susceptible to uncertainty in assessment.

Capturing richer information: On establishing the validity of an interval-valued survey response mode

“…Another practical concern remains in demonstrating the efficacy of this approach across a broader variety of real-world contexts-expanding upon a growing body of research (cf. Ellerby et al, 2019aEllerby et al, , 2020Navarro et al, 2016;Wallace et al, 2016). Finally, it is important that future empirical research is conducted to directly compare this interval-valued response mode against existing methods of both lower (e.g., Likert, VAS) and higher complexity (e.g., multi-step interval elicitation, SHELF, FRS)-to establish and quantify the trade-offs between information capture and important practical concerns regarding response efficiency (i.e., workload, survey duration, perceived complexity, training requirements), which have a large influence on likelihood of broader adoption.…”

“…Recent work by present authors has included development of open-source software for efficient digital administration of interval-valued surveys (DECSYS-c.f. Ellerby, McCulloch, Young & Wagner, 2019), which is available to download via https://www.lucidresearch.org/decsys.html from GitHub. This paper builds upon this-reporting a study designed to provide an initial validation of this approach.…”

Insights from interval-valued ratings of consumer products—a DECSYS appraisal

“…Complementary research studies are ongoing to address socio-technical considerations-concerning the basic efficacy of this response mode to capture multiple sources of respondent uncertainty and response variability-and also to demonstrate the value of the resulting data in real-world contexts [8]. The present study was conducted to demonstrate both the value provided by interval-valued responses and the use of the DECSYS software's ellipse response mode, to capture said intervals.…”

Do People Prefer to Give Interval-Valued or Point Estimates and Why?

“…This study provides evidence that interval elicitation using ellipses is not only effective (cf. [3]- [5]), but efficient. Results found that respondents preferred to give interval rather than point estimates, because they perceived them as no more difficult to provide, while permitting more effective communication of their desired responses.…”

“…Despite these, a growing body of complementary theoretical, practical and empirical work has contributed to a recent surge in interest in interval-valued response elicitation. This research has worked to address each of these barriers-ranging from empirical studies to establish efficacy and potential value across a variety of real-world applications [3]- [7], to creation of open-source software tools to facilitate administration at scale [8], to development and evaluation of mathematical and statistical methods to best handle and interpret the richer information that is captured [9]- [17], cf. Fig.…”