Active Learning of Markov Decision Processes using Baum-Welch algorithm

Bacci, Giovanni; Ingólfsdóttir, Anna; Larsen, Kim G.; Reynouard, Raphaël

doi:10.1109/icmla52953.2021.00195

Cited by 7 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Value Iteration (VI) was originally introduced to approximate expected rewards in MDPs [7]. In a broader sense, VI simply refers to iterating a function f : R |S| → R |S| (called Bellman operator in the MDP setting) from some given initial vector x (0) , i.e., to compute the sequence x (1) = f (x (0) ), x (2) = f (x (1) ), etc. Instances of VI are usually set up such that the sequence converges to a (generally non-unique) fixed point x = f (x).…”

Section: Value Iterationmentioning

confidence: 99%

“…EVTs are also employed in an algorithm proposed in [8] for LTL model checking of interval Markov chains. Moreover, EVTs have been leveraged for minimizing and learning DTMCs [1,2]. Further recent applications of EVTs to MDPs include verifying cause-effect dependencies [3], as well as an abstraction-refinement procedure that measures the importance of states based on the EVTs under a fixed policy [33].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Accurately Computing Expected Visiting Times and Stationary Distributions in Markov Chains

Mertens,

Katoen,

Quatmann

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We study the accurate and efficient computation of the expected number of times each state is visited in discrete- and continuous-time Markov chains. To obtain sound accuracy guarantees efficiently, we lift interval iteration and topological approaches known from the computation of reachability probabilities and expected rewards. We further study applications of expected visiting times, including the sound computation of the stationary distribution and expected rewards conditioned on reaching multiple goal states. The implementation of our methods in the probabilistic model checker scales to large systems with millions of states. Our experiments on the quantitative verification benchmark set show that the computation of stationary distributions via expected visiting times consistently outperforms existing approaches — sometimes by several orders of magnitude.

show abstract