Proceed with caution: using verbal protocol analysis to measure situation awareness

A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

“…Similarly, Golightly et al (2010) concluded SAGAT was not suited for SA measurement in the train driving domain, based on a short study in which they only collected a small amount of data at the end of the trial, significantly reducing its sensitivity. Rose et al (2019) found it sensitive in train driving, however.…”

“…Some researchers have examined the processes that people use to develop SA using measures such as eye tracking (de Winter, Eisma, Cabrall, Hancock, & Stanton, 2019; Ikuma, Harvey, Taylor, & Handal, 2014; Smolensky, 1993), communications (Bolstad et al, 2007; Gorman, Cooke, Pederson, Connor, & DeJoode, 2005; Orasanu, 2000; Prince, Salas, & Stout, 1995), verbal protocols (Hall & Phelps, 1983; Rose, Bearman, Naweed, & Dorrian, 2019; Sullivan & Blackman, 1991; Walker, Stanton, & Young, 2008), and physiological measurement (Wilson, 2000). While process measures can provide insights into how people develop SA, they can only be used to indirectly infer the quality and completeness of the resulting SA, as a state of knowledge about the situation, obtained by the individuals involved.…”

“…Furthermore, these techniques generally provide only partial insights into what information is attended to and how it is processed to form situation comprehension and projection. Verbal protocols have not been found to be effective for measuring SA (Rose et al, 2019; Walker et al, 2008), and physiological measures have had little research to establish validity for this purpose. Therefore, process measures are inadequate for measuring SA on their own.…”

Comparison of SAGAT and SPAM for Seeking Effective Way to Evaluate Situation Awareness and Workload During Air Traffic Control Task

“…In this period, an extra workload might be loaded to the participants because they should make effort to understand and predict past, present and future situations. Additionally, individual differences perhaps influenced participants' attitudes towards two methods (Rose et al, 2019). Further study should discuss the issue of SA evaluation technique with the working memory problem.…”

“…Additionally, Endsley (1988) investigated SA by asking participants the contents related to surrounding environments directly. However, the work of Rose et al (2019) described that because SA evaluation by an utterance is dependent upon individual differences in levels of knowledge or skills, individual SA between operators could not be compared easily. Additionally, judging based on operators' behavior and performance of task may be not facilitated accurately if the operators do not show performance as expected (Pritchett et al, 1995).…”