Learning ObjectivesUnderstand how Markov models can be used to analyze medical decisions and perform cost-effectiveness analysis.This case study introduces concepts that should improve understanding of the following:1. Markov models and their use in medical research. 2. Basics of health economics. 3. Replicating the results of a large prospective randomized controlled trial using a Markov Chain and Monte Carlo simulations, and 4. Relating quality-adjusted life years (QALYs) and cost of interventions to each state of a Markov Chain, in order to conduct a simple cost-effectiveness analysis.
IntroductionMarkov models were initially theroreticized at the beginning of the 20th century by Russian mathematician Andrey Markov [1]. They are stochastic processes that undergo transitions from one state to another. Over the years, they have found countless applications, especially for modeling processes and informing decision making, in the fields of physics, queuing theory, finance, social sciences, statistics and of course medicine. Markov models are useful to model environments and problems involving sequential, stochastic decisions over time. Representing such environments with decision trees would be confusing or intractable, if at all possible, and would require major simplifying assumptions [2]. Markov models can be examined by an array of tools including linear algebra (brute force), cohort simulations, Monte Carlo simulations and, for Markov Decision Processes, dynamic programming and reinforcement learning [3,4].