When detecting surface defects in the industrial cutting environment, the defects are easily contaminated and covered by many interference factors (such as chips and coolant residue) that exist on the machined surface. These interfering factors hinder the sustainable detection of surface defects. Furthermore, addressing the challenge of detecting surface defects in the presence of interference factors has proven to be a difficult problem in the current detection field. To solve this problem, a sustainable detection method for surface defects is proposed. The method is divided into two steps: one is the identification and removal of interference factors; the other is the detection of surface defects. First, a new FPN-DepResUnet model is constructed by modifying the Unet model from three aspects. The FPN-DepResUnet model is used to identify the interference factors in the image. Compared to the Unet model, the MAP of the FPN-DepResUnet model is increased by 5.77%, reaching 94.82%. The interfering factors are then removed using the RFR-net model. The RFR-net model performs point-to-point repair of interference regions. The repair process is performed by finding high-quality pixels similar to the interference region from the rest of the image. The negative effects of the interfering factors are removed by combining the FPN-DepResUnet model with the RFR-net model. On this basis, the SAM-Mask RCNN model is proposed for efficient defect detection of clean surface images. Compared with the Mask RCNN model, the MAP of the proposed SAM-Mask RCNN model increased by 2.00%, reaching 94.62%. Further, the inspection results can be fed back with a variety of surface defect information including defect types, the number of pixels in the different defect regions, and the proportion of different defect regions in the entire image. This enables predictive maintenance and control of the machined surface quality during machining.