The optimal level of fuzz: case studies in a methodology for psychological research

Abstract

Cognitive Science research is hard to conduct, because researchers must take phenomena from the world and turn them into laboratory tasks for which a reasonable level of experimental control can be achieved. Consequently, research necessarily makes tradeoffs between internal validity (experimental control) and external validity (the degree to which a task represents behavior outside of the lab). Researchers are thus seeking the best possible tradeoff between these constraints, which we refer to as the optimal level of fuzz. We present two principles for finding the optimal level of fuzz, in research, and then illustrate these principles using research from motivation, individual differences, and cognitive neuroscience.A hallmark of cognitive science involves the interplay of methods from different disciplines. Despite the importance of this interplay, methodological discussions in Psychology under the banner of cognitive science tend to focus on statistical issues such as the possibility that null hypothesis testing may lead research astray (e.g., Killeen, 2006). Much less discussion has centered on how to use the power of multidisciplinary cognitive science to construct research questions in ways that are likely to provide insight into the difficult questions that the field must address. In this paper, we present a principle that we call the optimal level of fuzz that we believe can guide good research. We start by defining the concept of fuzz and then discuss a set of principles that can guide researchers toward finding the optimal level of fuzz for their research. Next, we present three case studies of the optimal level of fuzz in action. Finally, we discuss the implications of this principle for research. What is Fuzz?Within cognitive science, experimental research in psychology provides data that can be used to constrain theories in neuroscience and psychology and to inspire new computational methods in artificial intelligence and reinforcement learning. Experimental research in psychology must typically trade off between internal and external validity. Internal validity is the basic idea that our experiments should be free from confounds and alternative explanations so that the results of our experiments can be unambiguously attributed to the variables we manipulated in our studies. External validity is the degree to which our studies reflect behavior that might actually occur outside the laboratory. laboratory to bring about desired behaviors and has constructed systems for illuminating internal mental processes. For example, many studies use lexical decision tasks in which a subject is shown strings of letters that may or may not form a word and is asked to judge as quickly as possible whether the string forms a word. This task is quite useful for measuring the activity of concepts during a cognitive process. Tasks like this have been used in a variety of studies ranging from work on language comprehension to studies of goal activation (e.g., Fishbach, Friedman, & Kruglanski, 2003;McNamara, 2005)...

The Motivation–Cognition Interface in Learning and Decision Making

“…Our research assumes that studying the influence of incentives on thinking requires experimental tasks for which the cognitive demands are well understood (Markman, Beer, Grimm, Rein, & Maddox, 2009). Critically, this involves the application of formal modeling techniques that allow the researcher to dissect the task and isolate the underlying cognitive processes (Maddox & Ashby, 1993, 2004.…”

Combining the strengths of naturalistic and laboratory decision-making research to create integrative theories of choice.

“…The dimensions of internal and external validity are independent in theory, but often correlated in practice (Markman et al, 2009). In order to achieve high levels of experimental control in a situation, researchers are typically forced to abstract away the complexity of the world in order to hold a number of factors constant and to manipulate others.…”

“…Often researchers working within an area accept the tradeoff between internal and external validity implicitly without thinking about ways to improve the quality of the tradeoff. In the past, my colleagues and I have argued that as our understanding of laboratory tasks improves, the models of these tasks can be used to create better descriptions of the performance of participants (Markman et al, 2009). Then, more complex aspects of performance can be incorporated into laboratory tasks and new studies will push forward the complexity of laboratory tasks to give them more external validity while maintaining internal validity.…”

Surprise, Recipes for Surprise, and Social Influence

“…Taking a broad view of surprise may appear at first to be a sloppy lumping together of different approaches to surprise. However, it might actually be a useful and appropriate degree of lumping, an "optimal level of fuzz" (Markman, Beer, Grimm, Rein, & Maddox, 2009) given the current goal of examining surprise and social influence.…”