Composite structures are widely used due to their excellent performance. To improve their safety and reliability, non-destructive testing (NDT) methods are implemented to achieve efficient damage detection. In this paper, a novel stress–strain-based damage detection approach is proposed for composite structures by using continuous wavelet transform (CWT) and selective kernel convolutional network (SKNet), which exhibit good robustness when dealing with the stress–strain signals collected from different positions of composite structures. First, a stress–strain-based measuring scheme is designed for vibration response monitoring of the composite structures. Second, the collected stress–strain signals are converted into two-dimensional time–frequency images by the CWT. Then, the SKNet is constructed to classify these images for damage detection. Finally, an experimental test is conducted on a composite capsule, data from which is employed to evaluate the proposed method. The comparison results of our proposed method and other state-of-the-art approaches demonstrate its superiority.