Tracking specific target is an important research area in machine vision. When tracking specific target encounters the same local features with light and dark variations, convolutional neural networks based on deep learning require large data sets to extract target feature information. To address these problems, this paper proposes a target of interest tracking scheme based on specific object imaging. The method allows for fast capture and processing of important information about the target of interest. In addition, small data sets are used to track the target and extract the target of interest efficiently. The target of interest can also be accurately tracked when the background and light changes. The results show that our method reduces the processing of redundant information, reduces the storage overhead, and tracks the target accurately.