Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning

Kallus, Nathan; Uehara, Masatoshi

doi:10.1287/opre.2021.2249

Cited by 39 publications

(39 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, if the amount of time necessary to collect outcome Y (t) in O(t) is long, then generating a long time series would take too much time to be practically useful. If one is interested in causal effects on a long term outcome and is willing to forgo utilizing known randomization probabilities for treatment, we advocate for the marginal target parameters as described in previous work by or Kallus & Uehara (2019).…”

Section: Discussionmentioning

confidence: 99%

“…where R is a second order remainder that is doubly-robust, with R( Q, Q0 , g, g 0,t ) = 0 if either Q = Q0 or g = g 0,t . 2018) and Kallus & Uehara (2019). We pursue the discussion on marginal parameters in more detail in subsection 8.1 in the appendix.…”

Section: Canonical Gradient and First Order Expansion Of The Target P...mentioning

confidence: 99%

“…We present below two alternative statistical parameters defined on the same statistical model as the parameter we consider in this article, and which were considered in previous works , Kallus & Uehara 2019. The parameters are marginal, as opposed to context-specific parameters we consider in the present article.…”

Section: Comparison With Marginal Parametersmentioning

confidence: 99%

“…Specifically, Ψ 1,1 (P ) = dP Co(1) (c o (1))Ψ co(1) (P ). Kallus and Uehara (2019) Let γ ∈ (0, 1). Kallus & Uehara (2019) consider the parameter…”

Section: Marginal Parameter By Van Der Laan Et Al (2018)mentioning

confidence: 99%

“…The work of van der Laan et al(2018) andKallus & Uehara (2019) studies marginal causal parameters, marginalizing over the distribution of C o (t), defined on the same statistical model as the parameter we consider in this article. In particular, van der Laan et al(2018) define target parameters and estimation of the counterfactual mean of a future (e.g., long term) outcome under a stochastic intervention on a subset of the treatment nodes, allowing for extensions to single unit causal effects.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Adaptive Sequential Design for a Single Time-Series

Malenica,

Bibaut,

van der Laan

2021

Preprint

View full text Add to dashboard Cite

The current work is motivated by the need for robust statistical methods for precision medicine; we pioneer the concept of a sequential, adaptive design for a single individual. As such, we address the need for statistical methods that provide actionable inference for a single unit at any point in time. Consider the case that one observes a single time-series, where at each time t, one observes a data record O(t) involving treatment nodes A(t), an outcome node Y (t), and time-varying covariates W (t). We aim to learn an optimal, unknown choice of the controlled components of the design in order to optimize the expected outcome; with that, we adapt the randomization mechanism for future time-point experiments based on the data collected on the individual over time. Our results demonstrate that one can learn the optimal rule based on a single sample, and thereby adjust the design at any point t with valid inference for the mean target parameter. This work provides several contributions to the field of statistical precision medicine. First, we define a general class of averages of conditional causal parameters defined by the current context ("context-specific") for the single unit time-series data. We define a nonparametric model for the probability distribution of the time-series under few assumptions, and aim to fully utilize the sequential randomization in the estimation procedure via the double robust structure of the efficient influence curve of the proposed target parameter. We present multiple exploration-exploitation strategies for assigning treatment, and methods for estimating the optimal rule. Lastly, we present the study of the data-adaptive inference on

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Canonical Gradient and First Order Expansion Of The Target P...mentioning

confidence: 99%

Section: Comparison With Marginal Parametersmentioning

confidence: 99%

“…Specifically, Ψ 1,1 (P ) = dP Co(1) (c o (1))Ψ co(1) (P ). Kallus and Uehara (2019) Let γ ∈ (0, 1). Kallus & Uehara (2019) consider the parameter…”

Section: Marginal Parameter By Van Der Laan Et Al (2018)mentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Adaptive Sequential Design for a Single Time-Series

Malenica,

Bibaut,

van der Laan

2021

Preprint

View full text Add to dashboard Cite

show abstract

Deep spectral Q‐learning with application to mobile health

Gao

Shi

Song

2023

Stat

View full text Add to dashboard Cite

Dynamic treatment regimes assign personalized treatments to patients sequentially over time based on their baseline information and time‐varying covariates. In mobile health applications, these covariates are typically collected at different frequencies over a long time horizon. In this paper, we propose a deep spectral Q‐learning algorithm, which integrates principal component analysis (PCA) with deep Q‐learning to handle the mixed frequency data. In theory, we prove that the mean return under the estimated optimal policy converges to that under the optimal one and establish its rate of convergence. The usefulness of our proposal is further illustrated via simulations and an application to a diabetes dataset.

show abstract

A review of recent advances in empirical likelihood

Liu

Zhao

2022

WIREs Computational Stats

View full text Add to dashboard Cite

Empirical likelihood is widely used in many statistical problems. In this article, we provide a review of the empirical likelihood method, due to its significant development in recent years. Since the introduction of empirical likelihood, variants of empirical likelihood have been proposed, and the applications of empirical likelihood in high dimensions have also been studied. It is necessary to summarize the new development of empirical likelihood. In this article, we give a review of the Bayesian empirical likelihood, the bias-corrected empirical likelihood, the jackknife empirical likelihood, the adjusted empirical likelihood, the extended empirical likelihood, the transformed empirical likelihood, the mean empirical likelihood, and the empirical likelihood with high dimensions. Finally, we have a brief survey of the computation and implementation for empirical likelihood methods.

show abstract

Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learning

Cited by 39 publications

References 23 publications

Adaptive Sequential Design for a Single Time-Series

Adaptive Sequential Design for a Single Time-Series

Deep spectral Q‐learning with application to mobile health

A review of recent advances in empirical likelihood

Contact Info

Product

Resources

About