Siamese Instance Search for Tracking

Abstract

In this paper 1 we present a tracker, which is radically different from state-of-the-art trackers: we apply no model updating, no occlusion detection, no combination of trackers, no geometric matching, and still deliver state-of-theart tracking performance, as demonstrated on the popular online tracking benchmark (OTB) and six very challenging YouTube videos. The presented tracker simply matches the initial patch of the target in the first frame with candidates in a new frame and returns the most similar patch by a learned matching function. The strength of the matching function comes from being extensively trained generically, i.e., without any data of the target, using a Siamese deep neural network, which we design for tracking. Once learned, the matching function is used as is, without any adapting, to track previously unseen targets. It turns out that the learned matching function is so powerful that a simple tracker built upon it, coined Siamese INstance search Tracker, SINT, which only uses the original observation of the target from the first frame, suffices to reach state-of-theart performance. Further, we show the proposed tracker even allows for target re-identification after the target was absent for a complete video shot. the learnt generic matching function, we present a tracker, which reaches state-of-the-art tracking performance. The presented tracker is radically different from state-of-the-art trackers. We apply no model updating, no occlusion detection, no combination of trackers, no geometric matching and alike. In each frame, the tracker simply finds the candidate patch that matches best to the initial patch of the target in the first frame by the learned matching function. Third, to learn the matching function, we use a two-stream Siamese network [3], which we design specifically for tracking. Further, in the absence of any drifting that one would expect by on-the-fly model updating, the proposed tracker allows for successful target object re-identification after the target was absent for a long period of time, e.g., a complete shot.

Tracking by Natural Language Specification

“…In [28] Nam et al enable fine-tuning to more than one domain, where each domain is represented by a single training sequence. In [34,3], tracking is cast as instance search for which a Siamese network architecture is used. The original window of the target is compared with candidates windows from the current frame by a similarity function, learned from many examples before the tracking starts.…”

“…The original window of the target is compared with candidates windows from the current frame by a similarity function, learned from many examples before the tracking starts. As they function on the similarity to a stable original, and they are not updated during the tracking, Siamese trackers achieve state-of-the-art performance and recovery from loss, while being robust against variations in the query definition [34].…”

“…We adopt the Siamese tracker in [34] as our starting point for the visual object tracking in our model, for its robustness against errors in the starting box-definition. Furthermore, the Siamese tracking scheme has the advantage that it does not rely on model dynamics and hence it can be rebooted at any time as long as the lingual description is valid.…”

“…Then, the discovered target is used as the visual specification for a visual tracker, e.g., [34,9]. Thus, the first step for model I and model II is the same, by applying the Lingual Specification Network on the first frame, namely Three models for our tracking by natural language specification.…”

“…The common approach is to specify a target by means of a bounding box around the object and to track this target as it moves throughout the video [38,33,20]. The paradigm has proven to be effective and considerable progress has been achieved [17,37,34,3,11]. Yet, the fundamental assumption of having a bounding box target specification available has never been challenged.…”