Deep Network Flow for Multi-object Tracking

Abstract

Data association problems are an important component of many computer vision applications, with multi-object tracking being one of the most prominent examples. A typical approach to data association involves finding a graph matching or network flow that minimizes a sum of pairwise association costs, which are often either hand-crafted or learned as linear functions of fixed features. In this work, we demonstrate that it is possible to learn features for network-flow-based data association via backpropagation, by expressing the optimum of a smoothed network flow problem as a differentiable function of the pairwise association costs. We apply this approach to multi-object tracking with a network flow formulation. Our experiments demonstrate that we are able to successfully learn all cost functions for the association problem in an end-to-end fashion, which outperform hand-crafted costs in all settings. The integration and combination of various sources of inputs becomes easy and the cost functions can be learned entirely from data, alleviating tedious hand-designing of costs.

Robust Multi-Modality Multi-Object Tracking

“…Recent research of MOT primarily follows the tracking-by-detection paradigm [6,11,38,50], where object of interests is first obtained by an object detector and then linked into trajectories via data association. The data association problem could be tackled from various perspectives, e.g., min-cost flow [11,20,37], Markov decision processes (MDP) [48], partial filtering [6], Hungarian assignment [38] and graph cut [44,49]. However, most of these methods are not trained in an end-to-end manner thus many parameters are heuristic (e.g., weights of costs) and susceptible to local optima.…”

“…However, most of these methods are not trained in an end-to-end manner thus many parameters are heuristic (e.g., weights of costs) and susceptible to local optima. To achieve end-to-end learning within the min-cost flow framework, Schulter et al [37] applies bi-level optimization by smoothing the linear programming and Deep Structured Model (DSM) [11] exploits the hinge loss. Their frameworks, however, are not designed for cross-modality.…”

“…Our mmMOT follows the tracking-by-detection paradigm to define the data association cost, which is solved as a min-cost flow problem [11,20,37]. Take the online MOT setting for example, assume there are N and M detections in two consecutive frames i and i + 1, denoted by X i = x i j | j = 1, · · · , N and X i+1 = x i+1 k | k = 1, · · · , M , respectively.…”

AlignNet-3D: Fast Point Cloud Registration of Partially Observed Objects

“…sequent data association [3,5,52,50,1,23]. Recent learning-based trends aim at learning data association affinity functions [47,39], at obtaining temporally-stable detections [10,44,19] and at tackling joint segmentation and tracking [44]. Even though high-level path planning requires motion perception in 3D space, the task of continuous 3D state estimation of tracked targets is often neglected in existing approaches.…”

“…Even though high-level path planning requires motion perception in 3D space, the task of continuous 3D state estimation of tracked targets is often neglected in existing approaches. Methods based on network flow [52,39,22,31] only focus on discrete optimization and do not estimate the continuous state of targets. Several approaches cast state estimation as inference in linear dynamical systems such as Kalman filters.…”