Shrinkage priors for isotonic probability vectors and binary data modeling, with applications to dose–response modeling

Abstract

Motivated by the need to model dose–response or dose‐toxicity curves in clinical trials, we develop a new horseshoe‐based prior for Bayesian isotonic regression modeling a binary outcome against an ordered categorical predictor, where the probability of the outcome is assumed to be monotonically non‐decreasing with the predictor. The set of differences between outcome probabilities in consecutive categories of the predictor is equipped with a multivariate prior having support over simplex. The Dirichlet distribution, which can be derived from a normalized sum of independent gamma‐distributed random variables, is a natural choice of prior, but using mathematical and simulation‐based arguments, we show that the resulting posterior is prone to underflow and other numerical instabilities, even under simple data configurations. We propose an alternative prior based on horseshoe‐type shrinkage that is numerically more stable. We show that this horseshoe‐based prior is not subject to the numerical instability seen in the Dirichlet/gamma‐based prior and that the horseshoe‐based posterior can estimate the underlying true curve more efficiently than the Dirichlet‐based one. We demonstrate the use of this prior in a model predicting the occurrence of radiation‐induced lung toxicity in lung cancer patients as a function of dose delivered to normal lung tissue. Our methodology is implemented in the R package isotonicBayes and therefore suitable for use in the design of dose‐finding studies or other dose–response modeling contexts.