A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

Abstract

Objective: To examine evidence of sensitivity, predictiveness, and methodological concerns regarding direct, objective measures of situation awareness (SA). Background: The ability to objectively measure SA is important to the evaluation of user interfaces and displays, training programs, and automation initiatives, as well as for studies that seek to better understand SA in both individuals and teams. A number of methodological criticisms have been raised creating significant confusion in the research field. Method: A meta-analysis of 243 studies was conducted to examine evidence of sensitivity and predictiveness, and to address methodological questions regarding Situation Awareness Global Assessment Technique (SAGAT), Situation Present Assessment Technique (SPAM), and their variants. Results: SAGAT and SPAM were found to be equally predictive of performance. SPAM (64%) and real-time probes (73%) were found to have significantly lower sensitivity in comparison to SAGAT (94%). While SAGAT was found not to be overly memory reliant nor intrusive into operator performance, SPAM resulted in problems with intrusiveness in 40% of the studies examined, as well as problems with speed-accuracy tradeoffs, sampling bias, and confounds with workload. Concerns about memory reliance, the utility of these measures for assessing Team SA, and other issues are also addressed. Conclusion: SAGAT was found to be a highly sensitive, reliable, and predictive measure of SA that is useful across a wide variety of domains and experimental settings. Application: Direct, objective SA measurement provides useful and diagnostic insights for research and design in a wide variety of domains and study objectives.

Gaps and Opportunities in Situational Awareness for Cybersecurity

“…Gutzwiller's research and our additional professional field observations in government, industry, and academia suggest that the utility of SA analysis and measurement has yet to be realized in cyberspace [Gutzwiller et al 2015], unlike in aviation, driving, process control and nuclear power plant control, healthcare, and interactions with autonomous systems [Endsley 2019]. With a few exceptions, we know of almost no experimental work on measuring or characterizing the process or product in developing cyber situational awareness from the human's perspective.…”

Can Survey Measures Predict Subsequent Key Performance Indicators of Maritime Safety? Confirmatory and Exploratory Analyses of the Association between Self - report and Objective Indices

“…As discussed above, previous research has associated SA for a given task with performance on that task (see e.g. Endsley, 2019). It is therefore reasonable to assume that crew members who report that individual and environmental factors allow them to perceive relevant safety elements (level 1 of SA), understand how they are connected (level 2 of SA), and anticipate what the elements and current situation could lead to (level 3 of SA) also perform their work more safely.…”

“…Various measurement approaches have been used to assess SA (Salmon, Stanton, Walker, & Green, 2006), and SA has been found to predict performance and decision making (Endsley, 2019;Strybel, Vu, Kraft, & Minakata, 2008). The concept of SA has been subject to debate (Dekker, Hummerdal, & Smith, 2010), and Dekker and Hollnagel (2004) suggested a shift from the focus on cognition to a focus on the products of cognition.…”

“…It would thus be resource extensive and unrealistic to measure the SA of all crew members across all their tasks. Previous studies have developed and supported an approach measuring the selfreported experience of SA across work settings (Hjellvik et al, 2019;Saetrevik, 2013;Saetrevik & Hystad, 2017, 2019. Measuring the crew's SA, as well as other safety factors "in the field" aboard CTVs provides information about the safety level aboard the vessels and the anticipated products of the crew's self-reported cognitive states and performance.…”

“…Self-report of safe or unsafe behaviour is often used as a proxy for actual behaviour (e.g. Hjellvik et al, 2019;Naevestad et al, 2019;Sneddon, Mearns, & Flin, 2013;Saetrevik & Hystad, 2017, 2019. The current study used self-report to measure the personal and collective safety responsibilities of the crew working on CTVs.…”