Decision Variance in Risk-Averse Online Learning

Abstract

Online learning has traditionally focused on the expected rewards. In this paper, a risk-averse online learning problem under the performance measure of the mean-variance of the rewards is studied. Both the bandit and full information settings are considered. The performance of several existing policies is analyzed, and new fundamental limitations on risk-averse learning is established. In particular, it is shown that although a logarithmic distribution-dependent regret in time T is achievable (similar to the risk-neutral problem), the worst-case (i.e. minimax) regret is lower bounded by Ω(T ) (in contrast to the Ω( √ T ) lower bound in the risk-neutral problem). This sharp difference from the risk-neutral counterpart is caused by the the variance in the player's decisions, which, while absent in the regret under the expected reward criterion, contributes to excess mean-variance due to the non-linearity of this risk measure. The role of the decision variance in regret performance reflects a risk-averse player's desire for robust decisions and outcomes.

Exploring the Tradeoff between Competitive Ratio and Variance in Online-Matching Markets

“…In that context, a common practice is to formulate the problem as one of the renowned Multi-armed bandit variants [40] and then conduct regret analysis, showing the expected total regret (defined as the gap in the total utility achieved by a given policy and a prophet optimal) is upper bounded by a certain function of the total time horizon [41][42][43][44]. A few recent works investigate the potential tradeoff between variance and regret in online learning; see, e.g., [45,46]. In particular, Vakili et al [46] introduced and analyzed the performance of several risk-averse policies in both bandit and full information settings under the metric of mean-variance [47].…”

“…A few recent works investigate the potential tradeoff between variance and regret in online learning; see, e.g., [45,46]. In particular, Vakili et al [46] introduced and analyzed the performance of several risk-averse policies in both bandit and full information settings under the metric of mean-variance [47].…”

A One-Size-Fits-All Solution to Conservative Bandit Problems

“…Mean-variance Bandit Literature. (Sani, Lazaric, and Munos 2012) open the mean-variance bandit literature which considers both the expected reward and variability into performance measures, and a series of follow-ups (Maillard 2013;Vakili, Boukouvalas, and Zhao 2019;Cardoso and Xu 2019) have emerged recently. To our best knowledge, this paper is the first to study the mean-variance bandit problem with conservative exploration.…”