Infrared object tracking plays a key role in many research fields, and there is a series of work on applying particle filter to this tracking problem. Most of the PF-based tracking algorithms utilize the Bhattacharyya coefficient as a similarity measure, however, its performance in infrared object tracking is limited due to insufficient discriminative power. In this paper, we present a combined similarity measure under the particle filter framework, which integrates the advantages of the Bhattacharyya coefficient, histogram intersection, and structural similarity. The experimental results are gained by using different infrared image sequences, which show that the proposed measure gives superior discriminative power and achieves more robust and stable tracking performance than the traditional approach.