A safety-critical decision support system evaluation using situation awareness and workload measures

Abstract

Abstract:To ensure the safety of operations in safety-critical systems, it is necessary to maintain operators' situation awareness (SA) at a high level. A situation awareness support system (SASS) has therefore been developed to handle uncertain situations [1]. This paper aims to systematically evaluate the enhancement of SA in SASS by applying a multi-perspective approach. The approach consists of two SA metrics, SAGAT and SART, and one workload metric, NASA-TLX. The first two metrics are used for the direct objective and subjective measurement of SA, while the third is used to estimate operator workload. The approach is applied in a safety-critical environment called residue treater, located at a chemical plant in which a poor human-system interface reduced the operator's SA and caused one of the worst accidents in US history. A counterbalanced within-subjects experiment is performed using a virtual environment interface with and without the support of SASS. The results indicate that SASS improves operators' SA, and specifically has benefits for SA levels 2 and 3. In addition, it is concluded that SASS reduces operator workload, although further investigations in different environments with a larger number of participants have been suggested.

The Divergence of Objective and Subjective Situation Awareness: A Meta-Analysis

“…Another study also showed mixed results, with one query positively correlated with SART and another negatively correlated (Saetrevik, 2012). The remaining eight studies found no significant correlation between these two measures (Endsley et al, 1998;Loft et al, 2015;Naderpour, Lu, & Zhang, 2016; Puuska et al, 2018;Salmon et al, 2009;Sorensen & Stanton, 2011;Strybel, Vu, Kraft, & Minakata, 2008;Walker, Stanton, & Young, 2006). Based on the six studies reporting correlation values, the mean correlation between SART and SAGAT was calculated as Pearson's r = .094, with a 95% confidence level between .013 and .174.…”

The future role of human operators in highly automated electric power systems

“…A comprehensive overview of metrics is provided by the Single European Sky ATM Research (SESAR) project in [37]. Two distinctive categories can be identified, subjective and objective [38], from which the following methods are deemed to be the most useful for modern ICT-driven systems [39].…”

“…[24,67,68,69]. Industrial automation in manufacturing and the process industry are perhaps the most advanced in terms of automation because of their long history [70,71,72,22,38]. Experiences and developments on SA in the automotive and aviation sectors also hold important findings [73,74,75].…”

Identification of occupational stress factors of commissioning workers in nuclear power plants based on a bottom-up survey design and factor analysis

“…Lai et al [18] found that exposure to normal intensity of psycho-physiological stress under high memory load rather than low memory load could demonstrate the beneficial effect of stress on response time without reducing accuracy through an improved Stenberg working memory test. Naderpour et al [19] found that in a safety-critical environment, a bad human-system interface could reduce the operators' situation awareness and increase their workload (essentially mental load) by applying situation awareness metric and NASA-TLX mental load metric.…”