Simple Artificial Neural Networks That Match Probability and Exploit and Explore When Confronting a Multiarmed Bandit

Abstract

Abstract-The matching law (Herrnstein 1961) states that response rates become proportional to reinforcement rates; this is related to the empirical phenomenon called probability matching (Vulkan 2000). Here, we show that a simple artificial neural network generates responses consistent with probability matching. This behavior was then used to create an operant procedure for network learning. We use the multiarmed bandit (Gittins 1989), a classic problem of choice behavior, to illustrate that operant training balances exploiting the bandit arm expected to pay off most frequently with exploring other arms. Perceptrons provide a medium for relating results from neural networks, genetic algorithms, animal learning, contingency theory, reinforcement learning, and theories of choice.Index Terms-Instrumental learning, multiarmed bandit, operant conditioning, perceptron, probability matching.

Get out of the corner: Inhibition and the effect of location type and number on perceptron and human reorientation

“…It is interesting to note that, ignoring features, the "correct" wall length configuration is reinforced on 50 % of its presentations (the reinforced location and its nonreinforced rotational equivalent), while the "incorrect" configurations present in any condition are reinforced 0 % of the time, and the perceptrons' responses converge to match these probabilities. The operant perceptron has already been established to match probabilities in classical choice-behavior tasks (Dawson et al, 2009); for it to exhibit this behavior in a reorientation context reinforces Miller and Shettleworth's (2007) conceptualization of reorientation as an operant task.…”

“…While we could extend the operant perceptron to see whether it fits the data from some of the novel tasks (i.e., regular octagons; Newcombe et al, 2010) in a manner similar to a more standard perceptron (Dawson, Kelly, et al, 2010), the synthetic approach would be to generate totally new predictions inspired by our findings. In this case, we might try nonuniform octagons (an arena type not yet investigated), or we might note other successes of the operant perceptron altogether, such as superconditioning (Dupuis & Dawson, in press) or probability matching (Dawson et al, 2009), and branch out beyond reorientation into completely new paradigms. Lewandowsky (1993) observed that computer modeling had its benefits, if done with care.…”

“…In effect, the perceptron will choose whether or not to visit a location with a probability based on how attractive the cues at that location are, and it will learn only from locations it chooses to visit. This algorithm is detailed at length in Dupuis and Dawson (in press) and in a more abbreviated form in Dawson et al (2009).…”

“…For locations that are reinforced, the perceptron is trained to turn on (output activity=1); for locations that are not reinforced, the perceptron is trained to turn off (output activity=0). Because, during learning, perceptron activity falls in the continuous range between 0 and 1, at any moment in time, the perceptron's output can be interpreted as its estimation of the conditional probability of reinforcement at a location, given the cues at that location (Dawson et al, 2009;Dupuis & Dawson, in press). …”

“…A perceptron trained with a standard learning algorithm has been shown to generate most of the interesting regularities found in reorientation task behavior (Dawson, Kelly, et al, 2010). An operant perceptron model of reorientation, which uses a more psychologically plausible learning algorithm (in which the perceptron has a chance of not visiting a location, based on the total associative strength at that location, and connection weights are adjusted only when the perceptron chooses to investigate a location), has also been proposed and has shown some promising results (Dawson, Dupuis, Spetch, & Kelly, 2009;Dupuis & Dawson, in press). The purpose of the present article is to illustrate how an operant perceptron can be used to explore reorientation by observing the model's behavior when novel reorientation paradigms are simulated.…”