Generating Reward Functions Using IRL Towards Individualized Cancer Screening

Petousis, Panayiotis; Han, Steve; Hsu, William; Bui, Alex A. T.

doi:10.1007/978-3-030-12738-1_16

Cited by 3 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When it comes to early cancer prediction (e.g., predicting screening t 2 cancer from screening t 0 ), the POMDP outperforms experts, indicating that the model and associated reward function are discriminating between positive and negative cases in a different way. This difference may be attributed to the dynamic observation model used with this POMDP; when independent observations are instead assumed, we have found kappa scores to 1 in other domains, indicating high correlation between the model and experts' decisions [12]. Indeed, error analysis of the POMDP's FPs shows a different subset from the physicians: cases with smaller nodule sizes but more years of smoking and older baseline age are predicted as false positives by the POMDP.…”

Section: Discussionmentioning

confidence: 83%

“…The POMDP we designed makes use of a reward function learned through analysis of physicians' past decisions. We recently presented an adaptive maximum entropy inverse reinforcement learning (MaxEnt IRL) algorithm to inform a reward function in different cancers [12]. Using MaxEnt IRL, we established an optimization function explicitly modeling experts' actions.…”

Section: Discussionmentioning

confidence: 99%

“…We learned a reward function using the recommendations of experts from the NLST dataset. Using inverse reinforcement learning (IRL), we learned state and action rewards via an adaptive maximum entropy IRL algorithm [12]. A multiplicative model was then employed to learn each combination of state-action pair rewards.…”

Section: ) Rewardsmentioning

confidence: 99%

“…Demonstrated in other settings [8]- [11], the implementation of POMDPs to guide clinical decision making is challenging given the need to derive required probability distributions and reward functions. We leveraged different techniques to learn a POMDP from NLST data: we integrated a dynamic Bayesian network (DBN) into the POMDP to predict the chance of developing lung cancer and to determine the POMDP's observation and transition probabilities, and we applied inverse reinforcement learning (IRL) to formulate a rewards model [12], mimicking experts' decisions.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Using Sequential Decision Making to Improve Lung Cancer Screening Performance

Petousis¹,

Winter

Speier

et al. 2019

IEEE Access

Self Cite

View full text Add to dashboard Cite

Globally, lung cancer is responsible for nearly one in five cancer deaths. The National Lung Screening Trial (NLST) demonstrated the efficacy of low-dose computed tomography (LDCT) to identify early-stage disease, setting the basis for widespread implementation of lung cancer screening programs. However, the specificity of LDCT lung cancer screening is suboptimal, with a significant false positive rate. Representing this imaging-based screening process as a sequential decision making problem, we combined multiple machine learning-based methods to learn a partially-observable Markov decision process that simultaneously optimizes lung cancer detection while enhancing test specificity. Using NLST data, we trained a dynamic Bayesian network as an observational model and used inverse reinforcement learning to discover a rewards function based on experts' decisions. Our resultant predictive model decreased the false positive rate while maintaining a high true positive rate at a level comparable to human experts. Our model also detected a number of lung cancers earlier. INDEX TERMS Early disease prediction, dynamic Bayesian networks, lung cancer screening, partially observable Markov decision processes, QMDP algorithm.

show abstract

Section: Discussionmentioning

confidence: 83%

Section: Discussionmentioning

confidence: 99%

Section: ) Rewardsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Using Sequential Decision Making to Improve Lung Cancer Screening Performance

Petousis¹,

Winter

Speier

et al. 2019

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…Yaylali and Karamustafa 13 modeled obesity levels based on BMI, cancer, and death to assess the impact of obesity on cancer and mortality using an MDP model. Petousis et al 14 used reversed enhancement learning to create reward commitments for Markov decision protocol of lung and breast cancer screening. Sun et al 15 evaluated the efficacy of breast cancer control compared to the lack of screening among women in urban and rural areas of China.…”

Section: Literature Reviewmentioning

confidence: 99%

An integrated methodology to control the risk of cardiovascular disease in patients with hypertension and type 1 diabetes

Shetaban

Esfahani

Saghaei

et al. 2020

Computational Intelligence

View full text Add to dashboard Cite

Today, air pollution, smoking, use of fatty acids and ready-made foods, and so on, have exacerbated heart disease. Therefore, controlling the risk of such diseases can prevent or reduce their incidence. The present study aimed at developing an integrated methodology including Markov decision processes (MDP) and genetic algorithm (GA) to control the risk of cardiovascular disease in patients with hypertension and type 1 diabetes. First, the efficiency of GA is evaluated against Grey Wolf optimization (GWO) algorithm, and then, the superiority of GA is revealed. Next, the MDP is employed to estimate the risk of cardiovascular disease. For this purpose, model inputs are first determined using a validated micro-simulation model for screening cardiovascular disease developed at Tehran University of Medical Sciences, Iran by GA. The model input factors are then defined accordingly and using these inputs, three risk estimation models are identified. The results of these models support WHO guidelines that provide medicine with a high discount to patients with high expected LYs. To develop the MDP methodology, policies should be adopted that work well despite the difference between the risk model and the actual risk. Finally, a sensitivity

show abstract

An Analytics‐Driven Approach for Optimal Individualized Diabetes Screening

Kamalzadeh

Ahuja

Hahsler

et al. 2021

Production and Operations Management

View full text Add to dashboard Cite

T ype 2 diabetes is a chronic disease that affects millions of Americans and puts a significant burden on the healthcare system. The medical community sees screening patients to identify and treat prediabetes and diabetes early as an important goal; however, universal population screening is operationally not feasible, and screening policies need to take characteristics of the patient population into account. For instance, the screening policy for a population in an affluent neighborhood may differ from that of a safety-net hospital. The problem of optimal diabetes screening-whom to screen and when to screen-is clearly important, and small improvements could have an enormous impact. However, the problem is typically only discussed from a practical viewpoint in the medical literature; a thorough theoretical framework from an operational viewpoint is largely missing. In this study, we propose an approach that builds on multiple methods -partially observable Markov decision process (POMDP), hidden Markov model (HMM), and predictive risk modeling (PRM). It uses available clinical information, in the form of electronic health records (EHRs), on specific patient populations to derive an optimal policy, which is used to generate screening decisions, individualized for each patient. The POMDP model, used for determining optimal decisions, lies at the core of our approach. We use HMM to estimate the cohort-specific progression of diabetes (i.e., transition probability matrix) and the emission matrix. We use PRM to generate observations-in the form of individualized risk scores-for the POMDP. Both HMM and PRM are learned from EHR data. Our approach is unique because (i) it introduces a novel way of incorporating predictive modeling into a formal decision framework to derive an optimal screening policy; and (ii) it is based on real clinical data. We fit our model using data on a cohort of more than 60,000 patients over 5 years from a large safety-net health system and then demonstrate the model's utility by conducting a simulation study. The results indicate that our proposed screening policy outperforms existing guidelines widely used in clinical practice. Our estimates suggest that implementing our policy for the studied cohort would add one quality-adjusted life year for every patient, and at a cost that is 35% lower, compared with existing guidelines. Our proposed framework is generalizable to other chronic diseases, such as cancer and HIV.

show abstract

Generating Reward Functions Using IRL Towards Individualized Cancer Screening

Cited by 3 publications

References 17 publications

Using Sequential Decision Making to Improve Lung Cancer Screening Performance

Using Sequential Decision Making to Improve Lung Cancer Screening Performance

An integrated methodology to control the risk of cardiovascular disease in patients with hypertension and type 1 diabetes

An Analytics‐Driven Approach for Optimal Individualized Diabetes Screening

Contact Info

Product

Resources

About