Sample path sharing in simulation-based policy improvement

Wu, Di; Jia, Qing‐Shan; Chen, Chun‐Hung

doi:10.1109/icra.2014.6907332

Cited by 3 publications

(5 citation statements)

References 43 publications

(24 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Herein, we compare our method with five existing methods (EA, OCBAPI [14], OCBA-S [15], EA-sample accumulation (SA), and OCBAPI-SA) using two MDP models, namely, a two-state example and its extended version. The extended version is used to verify the effectiveness and efficiency of the proposed method in a more complex manner.…”

Section: Methodsmentioning

confidence: 99%

“…a Sample path sharing is used to calculate QT,S π (s, a); see more details in [15]. The two-state example was expanded by increasing the available actions in state s 2 , as shown in Figure 3.…”

Section: Methodsmentioning

confidence: 99%

“…To solve the problem, the existing methods [14,15] apply OCBA to allocate a given simulation budget N efficiently; i.e., arg max {n 1 ,...n k ,T} P{CS},…”

Section: Problem Definitionmentioning

confidence: 99%

“…where π m PI represents the improved policy via mth SBPI. If N increases, the existing methods [14,15] can correctly select the best action in each state and improve the policy. However, before π converges to π * , the selected best action may not be the actual best action.…”

Section: Problem Definitionmentioning

confidence: 99%

“…Based on the merits of OCBA, Jia et al [14] applied it to improve the efficiency of SBPI and efficiently find the best action for a given state. Wu et al [15] developed a sample path sharing procedure to further improve upon the above work. For a given state, a sample path is obtained by selecting an action and thereafter following the given policy.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

An Efficient Simulation-Based Policy Improvement with Optimal Computing Budget Allocation Based on Accumulated Samples

Huang

2022

Electronics

View full text Add to dashboard Cite

Markov decision processes (MDPs) are widely used to model stochastic systems to deduce optimal decision-making policies. As the transition probabilities are usually unknown in MDPs, simulation-based policy improvement (SBPI) using a base policy to derive optimal policies when the state transition probabilities are unknown is suggested. However, estimating the Q-value of each action to determine the best action in each state requires many simulations, which results in efficiency problems for SBPI. In this study, we propose a method to improve the overall efficiency of SBPI using optimal computing budget allocation (OCBA) based on accumulated samples. Previous works have mainly focused on improving SBPI efficiency for a single state and without using the previous simulation samples. In contrast, the proposed method improves the overall efficiency until an optimal policy can be found in consideration of the state traversal property of the SBPI. The proposed method accumulates simulation samples across states to estimate the unknown transition probabilities. These probabilities are then used to estimate the mean and variance of the Q-value for each action, which allows the OCBA to allocate the simulation budget efficiently to find the best action in each state. As the SBPI traverses the state, the accumulated samples allow appropriate allocation of OCBA; thus, the optimal policy can be obtained with a lower budget. The experimental results demonstrate the improved efficiency of the proposed method compared to previous works.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

“…To solve the problem, the existing methods [14,15] apply OCBA to allocate a given simulation budget N efficiently; i.e., arg max {n 1 ,...n k ,T} P{CS},…”

Section: Problem Definitionmentioning

confidence: 99%

Section: Problem Definitionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations