Bandit online optimization over the permutahedron

Abstract

The permutahedron is the convex polytope with vertex set consisting of the vectors (π(1), . . . , π(n)) for all permutations (bijections) π over {1, . . . , n}. We study a bandit game in which, at each step t, an adversary chooses a hidden weight weight vector st, a player chooses a vertex πt of the permutahedron and suffers an observed instantaneous loss ofWe study the problem in two regimes. In the first regime, st is a point in the polytope dual to the permutahedron. Algorithm CombBand of Cesa-Bianchi et al (2009) guarantees a regret of O(n √ T log n) after T steps. Unfortunately, CombBand requires at each step an n-by-n matrix permanent computation, a #P -hard problem. Approximating the permanent is possible in the impractical running time of O(n 10 ), with an additional heavy inverse-polynomial dependence on the sought accuracy. We provide an algorithm of slightly worse regret O(n 3/2 √ T ) but with more realistic time complexity O(n 3 ) per step. The technical contribution is a bound on the variance of the Plackett-Luce noisy sorting process's 'pseudo loss', obtained by establishing positive semi-definiteness of a family of 3-by-3 matrices of rational functions in exponents of 3 parameters.In the second regime, st is in the hypercube. For this case we present and analyze an algorithm based on Bubeck et al.'s (2012) OSMD approach with a novel projection and decomposition technique for the permutahedron. The algorithm is efficient and achieves a regret of O(n √ T ), but for a more restricted space of possible loss vectors.

Structured Prediction with Projection Oracles

“…This is not surprising, as these choices do not produce valid permutation matrices. • Using a squared loss 1 2 ϕ(y) − θ 2 (C = R k×k , no projection) works relatively well when combined with permutation decoding. Using supersets of the Birkhoff polytope as projection set C, such as [0, 1] k×k or k×k , improves accuracy substantially.…”

“…The structured perceptron, hinge and CRF losses are generally not consistent when using MAP as decoder d [38]. Inspired by kernel dependency estimation [55,17,25], several works [15,26,31] showed good empirical results and proved consistency by combining a squared loss S sq (θ, y) := 1 2 ϕ(y) − θ 2 2 with calibrated decoding (no oracle is needed during training). A drawback of this loss, however, is that it does not make use of the output space Y during training, ignoring precious structural information.…”

Learning with Fenchel-Young Losses

“…It has been used for specific convex polytopes, most importantly in the optimal transport literature (Cuturi, 2013;Peyré and Cuturi, 2017) but also for learning to predict permutation matrices (Helmbold and Warmuth, 2009) or permutations (Yasutake et al, 2011;Ailon et al, 2016). The mean regularization counterpart of sparsemax is known as SparseMAP (Niculae et al, 2018):…”

“…Permutahedra have been used to derive online learning to rank algorithms (Yasutake et al, 2011;Ailon et al, 2016) but it is not obvious how to extract a loss from these works. Ordered weighted averaging (OWA) operators have been used to define related top-k multiclass losses (Usunier et al, 2009;Lapin et al, 2015) but without identifying the connection We set θ = y and inspect how the loss changes when varying each θ i .…”

Algorithmes de bandits pour la recommandation à tirages multiples

“…Certains algorithmes de bandits proposent de tirer les bras de manière ordonnée selon l'espérance estimée des bras par rapport à un utilisateur. Ailon et al (2014) propose l'algorithme BanditRank pour aborder la problématique d'ordonnancement. Pour cela l'algorithme soumet à chaque instant une permutation de l'ensemble des bras disponibles, dont l'objectif principal est de trier les bras par ordre de pertinence en fonction des récompenses obtenues.…”