Attention in Psychology, Neuroscience, and Machine Learning

Abstract

Attention is the important ability to flexibly control limited computational resources. It has been studied in conjunction with many other topics in neuroscience and psychology including awareness, vigilance, saliency, executive control, and learning. It has also recently been applied in several domains in machine learning. The relationship between the study of biological attention and its use as a tool to enhance artificial neural networks is not always clear. This review starts by providing an overview of how attention is conceptualized in the neuroscience and psychology literature. It then covers several use cases of attention in machine learning, indicating their biological counterparts where they exist. Finally, the ways in which artificial attention can be further inspired by biology for the production of complex and integrative systems is explored.

Deep Neural Network Models of Object Recognition Exhibit Human-Like Limitations when Performing Visual Search Tasks

“…In that regard, we point out that "selective attention" is often understood by researchers studying visual search as a source of performance limitations, while "attention" is usually understood by researchers in deep learning as some sort of learned weighting that results in performance improvements. Interdisciplinary studies reconciling these apparently conflicting definitions of attention might be fruitful (Lindsay, 2020;Marblestone et al, 2016), by using tasks like those we have presented here. In contrast to visual search, researchers studying object recognition assert that it occurs through an untangling mechanism (DiCarlo et al, 2012;DiCarlo & Cox, 2007).…”

“…These have also been described as models of visual selective attention (Eckstein, 1998; Geisler and Cormack, 2011; Wolfe and Horowitz, 2017; Peelen and Kastner, 2014). However, we defer usage of the term “attention” until the discussion, to place emphasis on which specific computations impose limits on search behavior (Hommel et al, 2019), and to avoid overloading the term, since “attention” has a different meaning when applied to the artificial neural network models that we test here (Lindsay, 2020; Hommel et al, 2019). The dominant models explaining limitations are built on results obtained by measuring search behavior with highly simplified stimuli like those shown in the upper left of Fig.…”

“…Visual search and object recognition are two real-world behaviors we engage in constantly, for example when we look for our keys on a cluttered desktop (Eckstein, 2011;Wolfe & Horowitz, 2017). Understanding how the brain solves the problems of object recognition and visual search is important for psychology, neuroscience, and artificial intelligence (DiCarlo et al, 2012;DiCarlo & Cox, 2007;Eckstein, 2011;Geisler & Cormack, 2011;Lindsay, 2020;Peelen & Kastner, 2014;Wolfe & Horowitz, 2017). Both behaviors have long been studied separately, but the two are obviously related: to find an object we are looking for, we must recognize it.…”

“…Thus, we chose the deep learning framework to test whether DNNs optimized for object recognition exhibit human-like limitations on performance across the different types of images used in visual search tasks: the highly-simplified stimuli used for controlled experiments, and relatively complex natural scenes containing real objects. Our focus on object recognition as a constraint on visual search is distinct from previous studies that have employed various neural network models to investigate visual search (Eckstein, 2011;Grossberg et al, 1994;Ma et al, 2011;Poder, 2017) and selective visual attention (Bobier et al, 2014;Lindsay, 2020;Lindsay & Miller, 2017). To measure the behavior of DNN-based object recognition models performing visual search tasks, we use methods from deep learning to adapt pre-trained models to new tasks.…”

Adaptive modulation and coding in underwater acoustic communications: a machine learning perspective

“…To address this issue, we resort to the attention mechanism and propose the A-kNN algorithm for the underwater AMC task. As a cognitive process of selectively concentrating on a few features while ignoring others, the attention mechanism can help ML models assign different weights to each part of the input, extract more critical and important information, and make more accurate judgments without incurring more costs to model computation and storage [30,31].…”