Efficient pest identification and control is critical for ensuring food safety. Therefore, automatic detection of pests has high practical value for Integrated Pest Management (IPM). However, complex field environments and the similarity in appearance among pests can pose a significant challenge to the accurate identification of pests. In this paper, a feature refinement method designed for similar pest detection in the field based on the two-stage detection framework is proposed. Firstly, we designed a context feature enhancement module to enhance the feature expression ability of the network for different pests. Secondly, the adaptive feature fusion network was proposed to avoid the suboptimal problem of feature selection on a single scale. Finally, we designed a novel task separation network with different fusion features constructed for the classification task and the localization task. Our method was evaluated on the proposed dataset of similar pests named SimilarPest5 and achieved a mean average precision (mAP) of 72.7%, which was better than other advanced object detection methods.