On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples

Thompson, William

doi:10.2307/2332286

Cited by 1,121 publications

(1,113 citation statements)

References 0 publications

Supporting

Mentioning

1,108

Contrasting

Unclassified

Order By: Relevance

“…This problem recapitulates the well-known explorationexploitation dilemma in reinforcement learning. To solve this problem our model selects strategies by Thompson sampling (May, Korda, Lee, & Leslie, 2012;Thompson, 1933) ( , problem) . This concludes the description of our model.…”

Section: A Rational Process Model Of Strategy Selectionmentioning

confidence: 99%

Strategy selection as rational metareasoning

Lieder¹,

Griffiths²

2016

Preprint

View full text Add to dashboard Cite

Many contemporary accounts of human reasoning assume that the mind is equipped with multiple heuristics that could be deployed to perform a given task. This raises the question of how the mind determines when to use which heuristic. To answer this question, we developed a rational model of strategy selection, based on the theory of rational metareasoning developed in the artificial intelligence literature. According to our model people learn to efficiently choose the strategy with the best cost-benefit tradeoff by learning a predictive model of each strategy's performance. We found that our model can provide a unifying explanation for classic findings from domains ranging from decisionmaking to arithmetic by capturing the variability of people's strategy choices, their dependence on task and context, and their development over time. Systematic model comparisons supported our theory, and four new experiments confirmed its distinctive predictions. Our findings suggest that people gradually learn to make increasingly more rational use of fallible heuristics. This perspective reconciles the two poles of the debate about human rationality by integrating heuristics and biases with learning and rationality.Keywords: bounded rationality; strategy selection; heuristics; meta-decisionmaking; metacognitive reinforcement learningTo succeed in life we have to solve a wide range of problems that place very different demands on us: sometimes we have to think fast and sometimes we have to think slow (cf. Kahneman, 2011). For instance, avoiding a car accident requires a split-second decision, whereas founding a successful start-up requires investing a lot of time into anticipating the future and weighting potential outcomes appropriately. No single decision mechanism works well across all situations. To meet the wide range of demands posed by different decision problems, it has been proposed that the human brain is equipped with multiple decision systems (Dolan & Dayan, 2013) and decision strategies (Payne, Bettman, & Johnson, 1988). Dual-process theories are a prominent example of this perspective (Evans & Stanovich, 2013;Evans, 2003;Kahneman, 2011). The coexistence of multiple alternative strategies is not specific to decision making. People also appear to possess multiple strategies for inference (Gigerenzer & Selten, 2002), memory (Bjorklund & Douglas, 1997), self-control (Braver, 2012), problem solving (Fum & Del Missier, 2001), and mental arithmetic (Siegler, 1999) to name just a few.The availability of multiple strategies that are applicable to the same problems raises the question how people decide when to use which strategy. The fact that so many different strategies have been observed under different circumstances shows that people's strategy choices are highly variable and contingent on the situation and the task (Beach & Mitchell, 1978;Fum & Del Missier, 2001;Payne, 1982;Payne et al., 1988). Overall, the contingency of people's strategy choices appears to be adaptive. Even though under certain circumstances p...

show abstract

Section: A Rational Process Model Of Strategy Selectionmentioning

confidence: 99%

Strategy selection as rational metareasoning

Lieder¹,

Griffiths²

2016

Preprint

View full text Add to dashboard Cite

show abstract

“…In this case, a Thompson Sampling algorithm [12] is applicable to the patrolling problem. As suggested in [13] [14], the robot based on this algorithm is enabled to find an optimal room with the highest expected intrusion probability.…”

Section: A Patrolling Strategy For Single Intrudermentioning

confidence: 99%

Patrolling robot based on Bayesian learning for multiple intruders

Hoshino

Ugajin

Ishiwata

2015

2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

View full text Add to dashboard Cite

Patrolling is an important task for autonomous mobile robots. In this paper, we focus on a single patrolling robot in indoor environments composed of rooms. For the robot, it is required to visit the rooms and detect unknown intruders as many as possible. However, the robot does not have correct information about intruders in the initiation step of patrolling. Therefore, the robot first acquires the information through the patrolling trials. As for the information, the robot focuses on intrusion probabilities into the rooms. Subsequently, target rooms are determined on the basis of the intrusion probabilities. This so-called explore/exploit tradeoff in reinforcement learning is a challenge for the intelligent patrolling. In this paper, we tackle the patrolling problem as a part of the learning issues. Thus a multi-armed bandit problem is introduced. For the challenge, we propose stochastic patrolling strategies based on Bayesian learning. Simulation experiments show that the robot is enabled to estimate the correct intrusion probabilities of the rooms. Through the discussion of the patrolling results, finally, we present the most effective patrolling strategy.

show abstract

“…This type of strategies keeps an estimate of the confidence interval related to each reward distribution and plays the arm with highest upper confidence bound at each time step. On the other hand, the Thompson Sampling class of algorithms introduces randomness on the exploration by sampling action parameters from their posterior distributions ( [7][8] [9]). …”

Section: Previous Workmentioning

confidence: 99%

Policies for Contextual Bandit Problems with Count Payoffs

Gisselbrecht¹,

Lamprier²,

Gallinari³

2015

2015 IEEE 27th International Conference on Tools With Artificial Intelligence (ICTAI)

View full text Add to dashboard Cite

The contextual bandit problem has been of major interest in the last few years. This corresponds to a sequential decision process where an agent has to choose at each iteration an action to perform, according to some knowledge about the decision environment and the current available actions, with the aim to maximize a cumulative amount of rewards over time. Many instances of the problem exist, depending on the kind of rewards we collect -real, binary, naturaland various algorithms are known to be efficient for some of these instances, either empirically or theoretically. In this paper we focus on the case of count payoffs, which corresponds to bandit problems where rewards are integer rewards, potentially unbounded. Based on a Bayesian Poisson regression model, we propose two new contextual bandit algorithms for this particular case with several concrete applications in real life: an Upper Confidence Bound algorithm and a Thompson Sampling strategy. Our approaches present the advantage to remain analytically tractable and computationally efficient. We experiment the algorithms on both simulated data and a real world scenario of spread maximization on a social network.

show abstract

On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples

Cited by 1,121 publications

References 0 publications

Strategy selection as rational metareasoning

Strategy selection as rational metareasoning

Patrolling robot based on Bayesian learning for multiple intruders

Policies for Contextual Bandit Problems with Count Payoffs

Contact Info

Product

Resources

About