In recent years, the foreign fibers in cotton lint significantly affect the quality of the final cotton textile products. It remains a challenging task to accurately distinguish foreign fibers from cotton. This article proposes an embedded system based on field programmable gate array (FPGA) þ digital signal processor (DSP) to recognize and remove foreign fibers mixed in cotton. With substantial tests of this system, we collect massive samples of foreign fibers and fake foreign fibers. Based on these samples, a convolution neural network mode is developed to validate the classification of the suspected targets from the detection subsystem, to improve the detection reliability. After training several model architectures, we find a model with the best balance between performance and computation. The high success rate (up to 96% in the validation set) demonstrates the effectiveness of the model. Moreover, the computation time (5 ms on a single image based on an eightcore DSP) indicates the efficiency of the detection, which ensures the real-time application of the system.