Statistical Power in Response Signal Paradigm Experiments

Logacev, Pavel; Bozkurt, M. İlteriş

doi:10.31234/osf.io/st6de

Cited by 3 publications

(4 citation statements)

References 8 publications

(13 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In consequence, the model-comparison metric adjusted R 2 that is typically used in the analysis of SAT data places an undue emphasis on the part of the SATF with the highest sampling variability. The result is high variability of by-subject parameter estimates and relatively low statistical power even for modest effect sizes (Logačev & Bozkurt, 2021; Pankratz et al, 2021).…”

Section: Methodsmentioning

confidence: 99%

The role of underspecification in relative clause attachment: Speed-accuracy tradeoff evidence.

Logacev¹

2023

Journal of Experimental Psychology: Learning, Memory, and Cogni

Self Cite

View full text Add to dashboard Cite

A number of studies have found evidence for the so-called ambiguity advantage, that is, faster processing of ambiguous sentences compared with unambiguous counterparts. While a number of proposals regarding the mechanism underlying this phenomenon have been made, the empirical evidence so far is far from unequivocal. It is compatible with several theories, including strategic underspecification (Swets et al., 2008), race models (Logačev & Vasishth, 2016; Van Gompel et al., 2000), and a more recent coactivation-based account (Dillon et al., 2019). While all three classes of theories make matching predictions for the average time to complete relative clause (RC) attachment in ambiguous compared with unambiguous sentences, their predictions diverge with regard to the minimum completion times. The speed-accuracy tradeoff procedure was used to test the predictions of all three classes of theories. According to a hierarchical Bayesian model, the speed-accuracy tradeoff functions (SATFs) for ambiguous RC attachment conditions show an earlier departure from chance performance than in either high or low attachment conditions. The results further indicate increased asymptotic accuracy in the ambiguous condition but no increase in processing rate. Taken together, this pattern of results is compatible with the strategic underspecification model and coactivation-based accounts, but not with race models.

show abstract

Section: Methodsmentioning

confidence: 99%

The role of underspecification in relative clause attachment: Speed-accuracy tradeoff evidence.

Logacev¹

2023

Journal of Experimental Psychology: Learning, Memory, and Cogni

Self Cite

View full text Add to dashboard Cite

show abstract

“…Results could well be different in, e.g., a single-response SAT study in which subjects respond only once per trial. Because no single-response SAT datasets are presently available, we have not analysed this paradigm here, though see Logačev and Bozkurt (2021), who show using simulated data that single-response SAT has somewhat higher power than multiple-response SAT.…”

Section: Power Analysis Of the Standard Sat Methodsmentioning

confidence: 99%

“…First, the usual method is based on non-hierarchical modelling, which fails to take all sources of variation into account. Second and more importantly, the statistical power of SAT designs has not yet been investigated (though see Logačev & Bozkurt, 2021). The latter issue is potentially a very serious one, since the arguments for the direct-access model depend on finding null results, but when statistical power is low, null results do not provide any information about whether the effect is actually absent (Greenland et al, 2016;Hoenig & Heisey, 2001).…”

Section: Introductionmentioning

confidence: 99%

Statistical properties of the speed-accuracy trade-off (SAT) paradigm in sentence processing

Pankratz¹,

Yadav²,

Smith³

et al. 2021

Preprint

View full text Add to dashboard Cite

Studies of the speed-accuracy trade-off (SAT) have been influential in arguing for the direct-access model of retrieval in sentence processing. The direct-access model assumes that long-distance dependencies (like subject-verb agreement) rely on a content-addressable search for the correct representation in memory. In this paper, we address two important weaknesses in the statistical methods standardly used for analysing SAT data. First, these methods are based on non-hierarchical modelling. We show how a hierarchical model can be fit to SAT data, and we test parameter recovery in this more conservative model. The parameters most relevant to the direct-access account cannot be accurately estimated. This may be due to the nature of SAT data or the standard SAT model. Second, the power properties of SAT studies are unknown. We conduct a power analysis and show that inferences from null results to the null hypothesis, though commonplace in the SAT literature, are likely unwarranted.

show abstract

“…As discussed in Vasishth & Gelman (2021), this has real, practical consequences for linguistics; if power is low, even if one repeatedly gets null results across multiple experiments, this does not imply that one has found evidence in favor of the null. The field is full of incorrect statistical inferences based on such null results from underpowered studies (e.g., Pankratz et al 2021;Logacev & Bozkurt 2021).…”

Section: Why Prospective Power Analysis Is So Importantmentioning

confidence: 99%

Some Right Ways to Analyze (Psycho)Linguistic Data

Vasishth

2023

Annu. Rev. Linguist.

View full text Add to dashboard Cite

Much has been written on the abuse and misuse of statistical methods, including p values, statistical significance, and so forth. I present some of the best practices in statistics using a running example data analysis. Focusing primarily on frequentist and Bayesian linear mixed models, I illustrate some defensible ways in which statistical inference—specifically, hypothesis testing using Bayes factors versus estimation or uncertainty quantification—can be carried out. The key is to not overstate the evidence and to not expect too much from statistics. Along the way, I demonstrate some powerful ideas, including the use of simulation to understand the design properties of one's experiment before running it, visualization of data before carrying out a formal analysis, and simulation of data from the fitted model to understand the model's behavior. Expected final online publication date for the Annual Review of Linguistics, Volume 9 is January 2023. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

show abstract

Statistical Power in Response Signal Paradigm Experiments

Cited by 3 publications

References 8 publications

The role of underspecification in relative clause attachment: Speed-accuracy tradeoff evidence.

The role of underspecification in relative clause attachment: Speed-accuracy tradeoff evidence.

Statistical properties of the speed-accuracy trade-off (SAT) paradigm in sentence processing

Some Right Ways to Analyze (Psycho)Linguistic Data

Contact Info

Product

Resources

About