Overrelaxed Sinkhorn–Knopp Algorithm for Regularized Optimal Transport

Abstract

This article describes a set of methods for quickly computing the solution to the regularized optimal transport problem. It generalizes and improves upon the widely used iterative Bregman projections algorithm (or Sinkhorn–Knopp algorithm). We first proposed to rely on regularized nonlinear acceleration schemes. In practice, such approaches lead to fast algorithms, but their global convergence is not ensured. Hence, we next proposed a new algorithm with convergence guarantees. The idea is to overrelax the Bregman projection operators, allowing for faster convergence. We proposed a simple method for establishing global convergence by ensuring the decrease of a Lyapunov function at each step. An adaptive choice of the overrelaxation parameter based on the Lyapunov function was constructed. We also suggested a heuristic to choose a suitable asymptotic overrelaxation parameter, based on a local convergence analysis. Our numerical experiments showed a gain in convergence speed by an order of magnitude in certain regimes.

A note on overrelaxation in the Sinkhorn algorithm

“…In this note we discuss a modified version of the Sinkhorn algorithm employing relaxation, which was recently proposed in [13] and [10]. It uses the update rule…”

“…After a similarity transform, this requires to compute the spectral norm of a symmetric matrix. An even simpler approach, as suggested in [13], is to directly estimate ϑ 2 , and hence ω opt , by monitoring the convergence rate of the standard Sinkhorn method in terms of a suitable residual. In the final section 4 we include numerical illustrations, which indicate that in certain cases such heuristics can be quite precise already in the initial phase of the algorithm, resulting in the almost optimal convergence rate.…”

“…However, the value of ω opt is a priori unknown in practice. Therefore we also tested a simple heuristic, similar to one suggested in [13]. It is known that the convergence of the Sinkhorn method can be monitored, e.g., through the error P 1 n − a 1 ; cf.…”

Low-rank tensor approximations for solving multi-marginal optimal transport problems

“…The pioneering work of Cuturi [14] established a relation between entropy regularized optimal transport and matrix scaling of exp(−C/η), where exp denotes the elementwise exponential and η > 0 denotes the regularization parameter. Scaling the rows and columns of exp(−C/η) such that marginal constraints are satisfied can be achieved numerically using the Sinkhorn algorithm [44], whose convergence speed can be accelerated using greedy coordinate descent [2,32], overrelaxation [45] or accelerated gradient descent [16]. This matrix scaling approach allows one to solve much larger optimal transport problems compared to previous attempts based on solving the original linear program, which in turn has impacted various fields including image processing [19,39], data science [38], engineering [34] and machine learning [22,30].…”

Unbalanced Optimal Transport through Non-negative Penalized Linear Regression

“…The introduction of this entropic regularization improves the scalability of OT, but involves a spreading of the mass and a loss of sparsity in the OT plan. When a sparse transport plan is sought, the convergence is slowed down, necessitating the use of acceleration strategies (Thibault et al, 2021). Regarding UOT with the (squared) 2 norm, Blondel et al (2018) showed that the resulting OT plan is sparse and proposed to use an efficient L-BFGS-B algorithm (Byrd et al, 1995) to address this case.…”