A grout sleeve connection is a typical kind of joint in prefabricated structures. However, for construction and manufacturing reasons, defects in this kind of joint are usually inevitable. The joint quality of a prefabricated structure has a significant influence on its overall performance and can lead to structural failure. Due to the complexity of various types of materials used in grout sleeve connections, traditional non-destructive testing methods, such as Acoustic Emission (AE), Ultrasonic Testing (UT), Guided Wave Testing (GW), are facing great challenges. The recent development of deep learning technology provides a new opportunity to solve this problem. Deep learning can learn the inherent rules and abstract hierarchies of sample data, and it has a powerful ability to extract the intrinsic features of training data in complex classification tasks. This paper illustrates a deep learning framework for the identification of joint defects in prefabricated structures. In this method, defect features are extracted from the acceleration time history response of a prefabricated structure using a convolutional neural network. The proposed method is validated by vibration experiments on a half-scaled, two-floor prefabricated frame structure with column rebars spliced by different defective grout sleeves.