Regret Circuits: Composability of Regret Minimizers

Abstract

Regret minimization is a powerful tool for solving large-scale problems; it was recently used in breakthrough results for large-scale extensiveform game solving. This was achieved by composing simplex regret minimizers into an overall regret-minimization framework for extensiveform game strategy spaces. In this paper we study the general composability of regret minimizers. We derive a calculus for constructing regret minimizers for composite convex sets that are obtained from convexity-preserving operations on simpler convex sets. We show that local regret minimizers for the simpler sets can be combined with additional regret minimizers into an aggregate regret minimizer for the composite set. As one application, we show that the CFR framework can be constructed easily from our framework. We also show ways to include curtailing (constraining) operations into our framework. For one, they enables the construction of CFR generalization for extensive-form games with general convex strategy constraints that can cut across decision points.

Combining No-regret and Q-learning

“…All three seem to rely on the same fundamental building block of better understanding the behavior of no-regret learners whose rewards are determined by (asynchronous) observations of other no-regret learners. Some recent progress along these lines has been made [20,33], but more work is needed.…”

“…CFR algorithms remain an active topic of research; recent work has shown how to combine it with function approximation [48,41,30,11,36], improve the convergence rate in certain settings [19], and apply it to more complex structures [20]. Most relevant to our work, examples are known where CFR fails to converge to the correct policy without perfect recall [35].…”

Convex-Concave Min-Max Stackelberg Games

“…Extensive-form games in which players' strategy sets can depend on other players' actions have been studied by Davis, Waugh, and Bowling [19] assuming payo s are bilinear, and by Farina, Kroer, and Sandholm [27] for another speci c class of convex-concave payo s.…”