Excess success in articles on object-based attention

Abstract

Twenty-five years of research has explored the object-based attention effect using the two-rectangles paradigm and closely related paradigms. While reading this literature, we noticed statistical attributes that are sometimes related to questionable research practices, which can undermine the reported conclusions. To quantify these attributes, we applied the Test for Excess Success (TES) individually to 37 articles that investigate various properties of object-based attention and comprise four or more experiments. A TES analysis estimates the probability that a direct replication of the experiments in a given article with the same sample sizes would have the same success (or better) as the original article. If the probability is low, then readers should be skeptical about the conclusions that are based on those experimental results. We find that 19 of the 37 analyzed articles (51%) seem too good to be true in that they have a replication probability below 0.1. In a new large sample study, we do find evidence for the basic object-based attention effect in the two-rectangles paradigm, which this literature builds on. A power analysis using this data shows that commonly used sample sizes in studies that investigate properties of object-based attention with the two-rectangles paradigm are, in fact, much too small to reliably detect even the basic effect.

Objects guide human gaze behavior in dynamic real-world scenes

“…Our modular and mechanistic framework establishes a connection between the role of objects in gaze guidance and ongoing discussions about effect sizes in object-based attention [ 150 ] or the behavior of the IOR across eye and object movements [ 53 ]. We chose to concentrate on the latter aspect due to the suitability of dynamic scenes.…”

Visual Attention in Crisis

“…Observers respond to a target about 10-20 ms faster when it appears on a cued object than when it appears on a different object, despite controlling for the distance between the invalid cue and the target (e.g. Egly, Driver, & Rafal, 1994;Francis & Thunell, 2022). This has been taken as evidence that the observer's attention automatically spreads from the cue location to the entire cued object.…”

Attending to Moving Objects

“…Based on the pattern of sample sizes, effect sizes, and p-values in three dozen published object-based attention studies, Francis and Thunell (2022) argued that publication bias and/or p-hacking in the literature is rife. Substantial proportions of researchers in psychology and other fields admit to such practices (John et al, 2012;Rabelo et al, 2020;Chin et al, 2021), and Francis and Thunell (2022) pointed out that the only previously-published study with a large sample (120 participants) found a non-significant effect, of only a 6 ms response time advantage (Pilz et al, 2012), and in Francis et al's own study with 264 participants, the effect was also quite small, at 14 ms. For an effect of this size, Francis et al calculated that the sample sizes typically used in the published literature were unlikely to yield statistical significance without some help from p-hacking or another questionable research practice. As a result, many papers in the literature make conclusions about objects and attention that unfortunately cannot be trusted.…”