Scalable grid‐based approximation algorithms for partially observable Markov decision processes

Kavaklioğlu, Can; Çevik, Mücahit

doi:10.1002/cpe.6743

“…We also extend our analysis to a previously developed POMDP model for cancer screening, which clearly demonstrate the benefits of distributed and parallel implementations. This work is covered in paper titled "Scalable gridbased approximation algorithms for partially observable Markov decision processes" published in Concurrency and Computation: Practice and Experience Journal (Kavaklioglu and Cevik, 2022).…”

Section: Rq: How Can We Design Scalable Pomdp Solution Algorithms Usi...mentioning

confidence: 99%

“…Lovejoy (1991a) showed how to use grid-based approximations to generate lower bounds and upper bounds on the optimal values obtained by the POMDPs. In a recent study, Kavaklioglu and Cevik (2022) compared distributed, parallel and sequential implementations of Lovejoy (1991a)'s lower bound and upper bound methods.…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

Novel Solution Methods for CPOMDPs With Applications to Cancer Screening

Kavaklioglu

2024

Preprint

0

View full text Add to dashboard Cite

<p>Partially Observable Markov Decision Processes (POMDPs) is a modeling framework that allow decision makers to make decisions in uncertain environments. This framework is particularly suited for sequential decision making problems for which the uncertainties are revealed in stages. Constrained POMDPs (CPOMDPs) extend this framework by also accounting for various restrictions in the environment. This PhD thesis focuses on POMDPs, CPOMDPs and its extensions, proposes novel solution methods and studies applications of POMDPs in disease screening. First, we focus on developing efficient versions of the well-known algorithms for POMDPs. POMDPs are notoriously difficult to solve optimally, and exact solution algorithms typically fail to solve the instances with more than a few core states. In this regard, we consider scalable algorithms solve the POMDP problems. Specifically, we propose distributed and parallel versions of the exact solution methods such as Monahan's algorithm and the incremental pruning algorithm as well as the heuristics such as Lovejoy's upper bound and lower bound methods. Next, we study the CPOMDPs and propose linear programming-based solution methods as a flexible approach for those. We perform a detailed numerical study with five different problem instances to showcase the viability of the proposed solution approach for CPOMDPs. Lastly, we propose multi-objective CPOMDP models for the breast cancer screening problem. These CPOMDP models extend the previously proposed POMDP models for breast cancer screening by considering different screening modalities and simultaneously optimizing for expected total quality-adjusted life years maximization and life-time risk minimization objectives.</p>

show abstract

“…We also extend our analysis to a previously developed POMDP model for cancer screening, which clearly demonstrate the benefits of distributed and parallel implementations. This work is covered in paper titled "Scalable gridbased approximation algorithms for partially observable Markov decision processes" published in Concurrency and Computation: Practice and Experience Journal (Kavaklioglu and Cevik, 2022).…”

Section: Rq: How Can We Design Scalable Pomdp Solution Algorithms Usi...mentioning

confidence: 99%

Novel Solution Methods for CPOMDPs With Applications to Cancer Screening

Kavaklioglu

2024

Preprint

0

View full text Add to dashboard Cite

<p>Partially Observable Markov Decision Processes (POMDPs) is a modeling framework that allow decision makers to make decisions in uncertain environments. This framework is particularly suited for sequential decision making problems for which the uncertainties are revealed in stages. Constrained POMDPs (CPOMDPs) extend this framework by also accounting for various restrictions in the environment. This PhD thesis focuses on POMDPs, CPOMDPs and its extensions, proposes novel solution methods and studies applications of POMDPs in disease screening. First, we focus on developing efficient versions of the well-known algorithms for POMDPs. POMDPs are notoriously difficult to solve optimally, and exact solution algorithms typically fail to solve the instances with more than a few core states. In this regard, we consider scalable algorithms solve the POMDP problems. Specifically, we propose distributed and parallel versions of the exact solution methods such as Monahan's algorithm and the incremental pruning algorithm as well as the heuristics such as Lovejoy's upper bound and lower bound methods. Next, we study the CPOMDPs and propose linear programming-based solution methods as a flexible approach for those. We perform a detailed numerical study with five different problem instances to showcase the viability of the proposed solution approach for CPOMDPs. Lastly, we propose multi-objective CPOMDP models for the breast cancer screening problem. These CPOMDP models extend the previously proposed POMDP models for breast cancer screening by considering different screening modalities and simultaneously optimizing for expected total quality-adjusted life years maximization and life-time risk minimization objectives.</p>

show abstract

“…A POMDP solver tool converts the POMDP model into a policy. There have been various algorithms for the solver ranging from the initial exact solution [8] to approximate solution methods like [ 10 , 11 ], and more recently [12] . Some algorithms use heuristic methods to arrive at a fast approximate solution [5] .…”

Section: Introductionmentioning

confidence: 99%