Robust Standards in Cognitive Science

Crüwell, Sophia; Stefan, Angelika; Evans, Nathan J.

doi:10.1007/s42113-019-00049-8

Cited by 34 publications

(50 citation statements)

References 80 publications

(136 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Subfields of psychology and neighbouring disciplines in which non-confirmatory research activities are common practice have already begun to tackle these issues (see e.g. Crüwell et al, 2019;Jacobs, 2020;Moravcsik, 2014). Drawing on existing expertise in these fields, exchanging resources, and starting broader discussions about underutilised methods may help us overcome our unhealthy fixation on hypothesis tests.…”

Section: Discussionmentioning

confidence: 99%

Why hypothesis testers should spend less time testing hypotheses

Scheel¹,

Tiokhin²,

Isager³

et al. 2020

Preprint

View full text Add to dashboard Cite

For almost half a century, Paul Meehl educated psychologists about how the mindless use of null-hypothesis significance tests made research on theories in the social sciences basically uninterpretable (Meehl, 1990). In response to the replication crisis, reforms in psychology have focused on formalising procedures for testing hypotheses. These reforms were necessary and impactful. However, as an unexpected consequence, psychologists have begun to realise that they may not be ready to test hypotheses. Forcing researchers to prematurely test hypotheses before they have established a sound ‘derivation chain’ between test and theory is counterproductive. Instead, various non-confirmatory research activities should be used to obtain the inputs necessary to make hypothesis tests informative. Before testing hypotheses, researchers should spend more time forming concepts, developing valid measures, establishing the causal relationships between concepts and their functional form, and identifying boundary conditions and auxiliary assumptions. Providing these inputs should be recognised and incentivised as a crucial goal in and of itself. In this article, we discuss how shifting the focus to non-confirmatory research can tie together many loose ends of psychology’s reform movement and help us lay the foundation to develop strong, testable theories, as Paul Meehl urged us to.

show abstract

Section: Discussionmentioning

confidence: 99%

Why hypothesis testers should spend less time testing hypotheses

Scheel¹,

Tiokhin²,

Isager³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…We note that our approach here is one of model application, rather than model validation or comparison (Crüwell, Stefan, & Evans, 2019). We assume that there is value in using the DMC as a theoretical framework and use its parameters to inform our question about whether common mechanisms exist.…”

Section: Overview Of the Papermentioning

confidence: 99%

Strategy and processing speed eclipse individual differences in control ability in conflict tasks

Hedge¹

2020

Preprint

View full text Add to dashboard Cite

The construct of response control or response inhibition is one of the cornerstones of modern cognitive psychology, featuring prominently in theories of executive functioning and impulsive behaviour. However, repeated failures to observe correlations between commonly applied tasks have led some theorists to question whether common or overlapping response conflict processes even exist. A challenge to answering this question is that behaviour is multifaceted, with both conflict and non-conflict processes (e.g. strategy, processing speed) contributing to individual differences. Here, we use a cognitive model to dissociate these processes; the diffusion model for conflict tasks (Ulrich et al., 2015). In a meta-analysis of fits to 7 empirical datasets, we observed weak (rho<.05) correlations between tasks in parameters reflecting conflict processing, seemingly challenging a general control construct. However, we saw consistent positive correlations in parameters representing processing speed and strategy. We then use model simulations to evaluate whether correlations in behavioural costs are diagnostic of the presence or absence of common mechanisms of conflict processing. We compare correlations in simulated behaviour in scenarios where we impose correlations in conflict parameters to scenarios in which only non-conflict parameters are correlated. We find that correlations in behaviour are neither necessary nor sufficient evidence for correlations in conflict parameters. Our data provide converging evidence to claims that non-conflict processes contribute substantially to individual differences in conflict tasks and illustrate that correlations between conflict tasks are only weakly informative about common conflict mechanisms.

show abstract

“…However, we wish to note that the aim of our study was only to distinguish between and compare di↵erent theories of speeded decision-making, and that our study does not have implications for the measurement properties of EAMs. Importantly, one of the most common uses of EAMs is as "measurement tools" of the decision-making process, where researchers use EAMs to estimate the latent parameters of the decision-making process -such as drift rate and threshold -and make inferences about how they vary across experimental conditions and/or groups (Crüwell et al, 2019). Having measurement tools that are theoretically accurate is important, as inaccuracies in the theoretical underpinning of the measurement tools may make the parameters estimated from them meaningless.…”

Section: Discussionmentioning

confidence: 99%

“…Importantly, the choice response time distributions alone were unable to qualitatively distinguish between these models, meaning that the additional constraint provided by the double response proportions created this distinction. However, our previous analyses only provided limited insight into why lateral inhibition helped in accounting for the double response proportions (see Evans, 2019c;Evans & Servant, 2020;Crüwell, Stefan, & Evans, 2019 for discussions on the importance of which and why in model assessment), with the only insight being that models without lateral inhibition provided a large over-prediction for the double response proportions, particularly for trials where an error was made for the initial response.…”

Section: Response Timementioning

confidence: 98%

Double responding: A new constraint for models of speeded decision making

Evans

Dutilh

Wagenmakers

et al. 2020

Cognitive Psychology

Self Cite

View full text Add to dashboard Cite

Evidence accumulation models (EAMs) have become the dominant models of speeded decision making, which are able to decompose choices and response times into cognitive parameters that drive the decision process. Several models within the EAM framework contain fundamentally di↵erent ideas of how the decision making process operates, though previous assessments have found that these models display a high level of mimicry, which has hindered the ability of researchers to contrast these di↵erent theoretical viewpoints.Our study introduces a neglected phenomenon that we term "double responding", which can help to further constrain these models. We show that double responding produces several interesting benchmarks, and that the predictions of di↵erent EAMs can be distinguished in standard experiment paradigms when they are constrained to account for the choice response time distributions and double responding behaviour in unison. Our findings suggest that lateral inhibition (e.g., the leaky-competing accumulator) provides models with a universal ability to make accurate predictions for these data. Furthermore, only models containing feed-forward inhibition (e.g., the di↵usion model) performed poorly under both of our proposed extensions of the standard EAM framework to double responding, suggesting a general inability of feed-forward inhibition to accurately predict these data.We believe that our study provides an important step forward in further constraining models of speeded decision making, though additional research on double responding is required before broad conclusions are made about which models provide the best explanation of the underlying decision-making process.

show abstract

Robust Standards in Cognitive Science

Cited by 34 publications

References 80 publications

Why hypothesis testers should spend less time testing hypotheses

Why hypothesis testers should spend less time testing hypotheses

Strategy and processing speed eclipse individual differences in control ability in conflict tasks

Double responding: A new constraint for models of speeded decision making

Contact Info

Product

Resources

About