Multi-objective multi-armed bandit with lexicographically ordered and satisficing objectives

Abstract

We consider multi-objective multi-armed bandit with (i) lexicographically ordered and (ii) satisficing objectives. In the first problem, the goal is to select arms that are lexicographic optimal as much as possible without knowing the arm reward distributions beforehand. We capture this goal by defining a multi-dimensional form of regret that measures the loss due to not selecting lexicographic optimal arms, and then, propose an algorithm that achieves Õ(T 2∕3 ) gap-free regret and prove a regret lower bound of (T 2∕3 ) . We also consider two additional settings where the learner has prior information on the expected arm rewards. In the first setting, the learner only knows for each objective the lexicographic optimal expected reward. In the second setting, it only knows for each objective a near-lexicographic optimal expected reward. For both settings, we prove that the learner achieves expected regret uniformly bounded in time. Then, we show that the algorithm we propose for the second setting of lexicographically ordered objectives with prior information also attains bounded regret for satisficing objectives. Finally, we experimentally evaluate the proposed algorithms in a variety of multi-objective learning problems.

Multiobjective Lipschitz Bandits under Lexicographic Ordering

“…To deal with these real-world applications, a natural idea is to utilize lexicographic ordering, as it ranks the objectives according to their importance (Ehrgott 2005;Wray, Zilberstein, and Mouaddib 2015;Hüyük and Tekin 2021;Hosseini et al 2021;Skalse et al 2022). Let X represent an arm space, and the expected payoffs for a, b ∈ X are [µ 1 (a), µ 2 (a), .…”

“…, m}, such that µ i (a) = µ i (b) for 1 ≤ i ≤ i * − 1 and µ i * (a) > µ i * (b). The lexicographically optimal arm is the one that is not lexicographically dominated by any other arms (Hüyük and Tekin 2021).…”

“…The only existing algorithm for multiobjective bandits under lexicographic ordering is specifically designed for the MOMAB model (Hüyük and Tekin 2021), whose arm set is finite, i.e., X = [K] 1 . Let x * denote the lexicographically optimal arm among X and x t be the arm chosen at t-th epoch.…”

“…Let x * denote the lexicographically optimal arm among X and x t be the arm chosen at t-th epoch. Hüyük and Tekin (2021) defined a priority-based regret to evaluate the performance of their algorithm, given by…”

“…The Thirty-Eighth AAAI Conference on Artificial Intelligence event that the previous i − 1 expected payoffs of the chosen arm are optimal, i.e., A i (x t ) = {µ j (x * ) − µ j (x t ) = 0, j ∈ [i−1]}. Hüyük and Tekin (2021) proposed an algorithm with a bound of O((KT ) 2/3 ) under this priority-based regret.…”