This paper considers target tracking as a binary classification problem to label pixels as either belonging to the target or the background. We present a novel robust algorithm, the multi-feature based ensemble classification and regression tree (ECART), for target tracking in infrared imagery (IR). In the first frame, a region of interest (ROI) containing target and background is initialized manually. Based on the multiple features of pixels, the ECART is trained online to distinguish between the target and the background. In the subsequent frames, the position and size of the ROI are predicted by the position and size of the target in the previous frame, respectively. Then, the ECART is used to label pixels within the predicted ROI, giving the label map. The new position and size of the target are finally found in the label map. Experimental results indicate that the proposed algorithm is effective and robust.