Efficient Linear Programming for Dense CRFs

Abstract

The fully connected conditional random field (CRF) with Gaussian pairwise potentials has proven popular and effective for multi-class semantic segmentation. While the energy of a dense CRF can be minimized accurately using a linear programming (LP) relaxation, the state-of-the-art algorithm is too slow to be useful in practice. To alleviate this deficiency, we introduce an efficient LP minimization algorithm for dense CRFs. To this end, we develop a proximal minimization framework, where the dual of each proximal problem is optimized via block coordinate descent. We show that each block of variables can be efficiently optimized. Specifically, for one block, the problem decomposes into significantly smaller subproblems, each of which is defined over a single pixel. For the other block, the problem is optimized via conditional gradient descent. This has two advantages: 1) the conditional gradient can be computed in a time linear in the number of pixels and labels; and 2) the optimal step size can be computed analytically. Our experiments on standard datasets provide compelling evidence that our approach outperforms all existing baselines including the previous LP based approach for dense CRFs.

Optimization and Machine Learning

“…The remaining of this section contains a paper published at CVPR 2017, namely [44]. This work was done in collaboration with Thalaiyasingam Ajanthan which worked on the proximal minimization algorithm for the LP relaxation of Dense CRFs of Section 2.2.4.…”

Adaptive appearance separation for interactive image segmentation based on Dense CRF

“…Desmaison et al [12] demonstrate the benefits of continuous relaxations over the mean‐field inference and provide an excellent alternative solution to the Dense CRF framework. Ajanthan et al [13] propose an efficient linear programming algorithm for minimising the potential of Dense CRF. Their method can achieve a potential drop in a relatively short period of time, however, they also point out that a lower potential does not mean a better segmentation result.…”

Efficient Relaxations for Dense CRFs with Sparse Higher-Order Potentials

“…The energy function of the CRF consists of a sum of three types of terms: unary energies that depend on the label for one random variable; pairwise energies that depend on the labels of two random variables; and higher-order energies that depend on a collection of random variables. Notable works such as [2,10,19] focus on just the unary and pairwise energies, leaving out the higher-order energies for computational efficiency.…”

“…Specifically, we formulate the energy function as both a QP and a LP relaxation and go on to show that both can be optimised efficiently using the filter-based method. As a novel contribution, we then extend the energy minimisation algorithms of our existing work [2,10] to deal with these higher-order potentials, whilst maintaining a complexity that is linear in the number of labels and random variables at each iteration.…”

“…There are preliminary versions of this work available, and the interested reader is encouraged to visit [2,10]. This work can be considered as a unified view of our previous work [2,10] with a novel addition of higher-order potentials. To keep this paper self-contained all relevant information and findings are detailed in this paper.…”