Decision Strategies Determined From the Shape of the Same–different Roc Curve: What Are the Effects of Incorrect Assumptions?

Abstract

The literature includes many studies that use the shape of the receiver operating characteristic (ROC) curve derived from the same–different test to ascertain the decision strategy used by the judge: a symmetrical ROC curve indicates a β ‐decision strategy, and an asymmetrical ROC curve indicates a τ ‐decision strategy. These studies have not considered the effect that breaches of the assumptions that underlie the standard same–different model would have on inferences drawn from the shape of the ROC curve. Three such assumptions are investigated: the equal‐variance assumption; the assumption of no stimulus order effects; and the assumption that the discriminability of the two stimuli is the same when presented first or second in the test sequence. Simulations indicate that the shape of the ROC curve is very tolerant of the breach of these assumptions. Consequently, inferences about the decision strategy used by a judge are fairly robust, and confidence in published results is upheld. PRACTICAL APPLICATIONS The same–different test is frequently used to determine the decision strategy adopted by judges. There are several assumptions made about the same–different test that may, if they were breached, undermine this test's ability to indicate the judge's decision strategy. The simulations conducted in this research indicate that the shape of the same–different ROC curve is very robust to breaches of these assumptions. Consequently, food scientists can maintain their confidence in the inferences made in the research already published on this topic, and in their own future research and development of sensory testing methods.

Affective same-different discrimination tests for assessing consumer discriminability between milks with subtle differences

“…There are two major decision strategies used in the process of the same-different judgments: the s-decision strategy, and the b-decision strategy. Although the results obtained from same-different tests with simple stimuli are commonly analyzed using s-strategy, several studies have reported that a b-strategy can also be used for same-different tests for food discrimination (Lee et al, 2007;Lee, van Hout, & Hautus, 2007;Hautus, O'Mahony, & Lee, 2008). Thus it is important to uncover the decision strategy that is being used in the present consumer same-different discrimination tests.…”

“…The application of the ROC curve for analysis in food sensory science is reviewed in and the studies that used the shape of the ROC curve derived from the same-different test to investigate the decision strategy used by the subjects were reviewed in Hautus, O'Mahony, and Lee (2008). To determine the best fitted strategy (model) for the ROC curve, the goodness-of-fit statistic (v 2 ) and the probability that the data arose from the strategy fitted given that the strategy was correct (p) were noted.…”

ROC curve analysis to determine effects of repetition on the criteria for same–different and A Not-A tests

“…It is not exactly the same shape as the equal variance s-criterion curve, but it is close (Hautus, O'Mahony, & Lee., 2008;O'Mahony and Hautus, 2008). It is logically possible, although unlikely, with the ROC analysis currently being used, that detection of a s-strategy (equal variance) might be a false result; it might be the detection of a b-strategy with unequal variance.…”

“…Irwin, Hautus, and Stillman (1992) reviewed ROC curves (Green & Swets, 1974) and some of the drawbacks of R-Index measurement for methods that induced a b-criterion. O'Mahony and and Hautus, O'Mahony and Lee (2008) reviewed the use of ROC curves and described how their shapes could be used to indicate whether judges were using a cognitive strategy that involved a b-or a s-criterion. Irwin et al (1993) demonstrated how ROC curves, derived from same-different tests for orange drinks, were best fitted assuming a cognitive strategy that used a s-criterion.…”

Decision strategies for the two-alternative forced choice reminder paradigm

“…While this is an unlikely experiment from the perspective of a psychophysicist, experiments of this kind are conducted frequently in the area of sensory evaluation. The goal here is to establish the sensitivity of the observer to stimuli (usually food products or beverages) under the normal conditions of consumption (see Hautus, O'Mahony, & Lee, 2008, for a description of this usage). It is in this context that the duo-trio procedure is frequently employed, and further research is required to determine whether or not observers use the comparison-of-distances strategy in this context.…”