Bridge expansion joints (BEJs) in service are susceptible to damage from various factors such as fatigue, impact, and environmental conditions. While visual inspection is the most common approach for inspecting BEJs, it is subjective and labor-intensive. In this paper, we propose a novel methodology for detecting the fault status of BEJs, inspired by voiceprint recognition (VPR) based on audio signals. We establish an Artificial Neural Network to filter nonevent segments from low signal-to-noise ratio signals, achieving an AuC value of 0.981. We design and improve ConFormer VPR models with a multifeature aggregation strategy and cascade them to realize fault detection of BEJs. For three successive tasks in classifying environment sound types, vehicle impact types, and faults, the ConFormer VPR models achieve AuC values of 0.975, 0.925, and 0.886, respectively, demonstrating the feasibility of our methods for unmanned inspection of BEJs. In future research, the introduction of multiple types of damage and the implementation of benchmarking tests are planned to further enhance the capabilities of the system.