The Differentiable Cross-Entropy Method

Abstract

We study the Cross-Entropy Method (CEM) for the non-convex optimization of a continuous and parameterized objective function and introduce a differentiable variant (DCEM) that enables us to differentiate the output of CEM with respect to the objective function's parameters. In the machine learning setting this brings CEM inside of the end-to-end learning pipeline where this has otherwise been impossible. We show applications in a synthetic energy-based structured prediction task and in non-convex continuous control. In the control setting we show on the simulated cheetah and walker tasks that we can embed their optimal action sequences with DCEM and then use policy optimization to fine-tune components of the controller as a step towards combining model-based and model-free RL.

Learning Convex Optimization Models

“…Recently, [2,6] showed how to efficiently differentiate through convex cone programs by applying the implicit function theorem to a residual map introduced in [27], and [1] showed how to differentiate through convex optimization problems by an automatable reduction to convex cone programs; our method for learning convex optimization models builds on this recent work. Optimization layers have been used in many applications, including control [7,11,15,3], game-playing [46,45], computer graphics [37], combinatorial tasks [58,52,53,21], automatic repair of optimization problems [14], and data fitting more generally [9,17,16,10]. Differentiable optimization for nonconvex problems is often performed numerically by differentiating each individual step of a numerical solver [33,48,32,36], although sometimes it is done implicitly; see, e.g., [7,47,4].…”

NOVAS: Non-convex Optimization via Adaptive Stochastic Search for End-to-End Learning and Control

“…Relation to Differentiable Cross-Entropy: Particular importance should be given to a recent paper [9], since, to the best of our knowledge, is the first to suggest sampling-based optimization instead of gradient descent, and features some similarities with our approach. The authors in [9] propose a differentiable approximation of the cross-entropy method (CEM) [21,22], called differentiable cross-entropy (DCEM). To obtain this approximation, they need to approximate CEM's eliteness threshold operation, which is non-differentiable.…”

“…With respect to the latter, a significant amount of attention has been devoted to incorporating optimization blocks or modules operating at some part of the network. This has been motivated by large number of applications, including meta-learning [1][2][3], differentiable physics simulators [4], classification [5], GANs [6], reinforcement learning with constraints, latent spaces, or safety [7][8][9][10], model predictive control [11,12], as well as tasks relying on the use of energy networks [13,3], among many others. Local 2 optimization modules lead to nested optimization operations, as they interact with the global, end-to-end training of the network that contains them.…”

“…One drawback in using this unrolled gradient descent operation however is the fact that doing so can lead to over-fitting to the selected gradient descent hyper-parameters, such as learning rate and number of iterations. Recently, a paper demonstrated promising results in alleviating this phenomenon by replacing these iterations of gradient descent by iterations of sampling-based optimization, in particular a differentiable approximation of the cross-entropy method [9]. While still unrolling the graph created by the fixed number of iterations, [9] showed empirically that no overfitting to the hyper-parameters occurred by performing inference on the trained network with altered inner-loop optimization hyper-parameters.…”

“…Recently, a paper demonstrated promising results in alleviating this phenomenon by replacing these iterations of gradient descent by iterations of sampling-based optimization, in particular a differentiable approximation of the cross-entropy method [9]. While still unrolling the graph created by the fixed number of iterations, [9] showed empirically that no overfitting to the hyper-parameters occurred by performing inference on the trained network with altered inner-loop optimization hyper-parameters. Another significant bottleneck in all methods involving graph unrolling is the number of iterations, which has to be kept low to prevent a prohibitively large graph during backprop, which is hard to train.…”