Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL

Abstract

Artificial behavioral agents are often evaluated based on their consistent behaviors and performance to take sequential actions in an environment to maximize some notion of cumulative reward. However, human decision making in real life usually involves different strategies and behavioral trajectories that lead to the same empirical outcome. Motivated by clinical literature of a wide range of neurological and psychiatric disorders, we propose here a more general and flexible parametric framework for sequential decision making that involves a two-stream reward processing mechanism. We demonstrated that this framework is flexible and unified enough to incorporate a family of problems spanning multi-armed bandits (MAB), contextual bandits (CB) and reinforcement learning (RL), which decompose the sequential decision making process in different levels. Inspired by the known reward processing abnormalities of many mental disorders, our clinically-inspired agents demonstrated interesting behavioral trajectories and comparable performance on simulated tasks with particular reward distributions, a real-world dataset capturing human decision-making in gambling tasks, and the PacMan game across different reward stationarities in a lifelong learning setting (The codes to reproduce all the experimental results can be accessed at https://github.com/doerlbh/mentalRL.).

Understanding Human Decision Processes: Inferring Decision Strategies From Behavioral Data

“…Human behavior inference using reinforcement learning (RL) is particularly common in robotics and contemporary neuroscience. Lin et al (2019) used a model free RL approach to understanding decision making. However, this study focused on abnormal processes in psychiatry.…”

Regularity Normalization: Neuroscience-Inspired Unsupervised Attention across Neural Network Layers

“…Draw attention to neuroscience-inspired algorithms: Like deep learning, many fields in artificial intelligence (AI) benefited from a rich source of inspirations from the neuroscience for architectures and algorithms. While the flow of inspirations from neuroscience to machine learning has been sporadic [ 50 ], it systematically narrows the major gaps between humans and machines: the size of required training datasets [ 51 ], out-of-set generalization [ 52 ], adversarial robustness [ 53 , 54 ], reinforcement learning [ 13 , 55 , 56 ] and model complexity [ 57 ]. Biological computations which are critical to cognitive functions are usually excellent candidates for incorporation into artificial systems and the neuroscience studies can provide validation of existing AI techniques for its plausibility as an integral component of an overall general intelligence system [ 58 ].…”

Modular deep reinforcement learning from reward and punishment for robot navigation

“…RP-AC [16] is a revamped reward-punishment Actor-Critic framework that formulates a policy gradient for continuous control. Split-Q Learning [17] suggested a more generalized reward-punishment framework by parameterizing immediate positive and negative rewards and their approximated state-action-values, aligning with various neurological and psychiatric mechanisms.…”