Achieving accurate and efficient detection of road damage in complex scenes has always been a challenging task. In this paper, an enhanced lightweight network, E-EfficientDet, is proposed. Firstly, a feature extraction enhancement module (FEEM) is designed to increase the receptive field and improve the feature expression capability of the network, which can extract richer multi-scale feature information. Secondly, to promote the reuse of feature information between different layers in the network and take full advantage of multi-scale context information, four pyramid modules with different structures are designed based on the idea of semi-dense connection, among which the bidirectional feature pyramid network with longitudinal connection (LC-BiFPN) is more suitable for road damage detection. Finally, to meet the road damage detection tasks under different hardware resource constraints, the E-EfficientDet-D0~D2 networks are proposed in this paper based on the compound scaling strategy. Experimental results show that the detection accuracy of E-EfficientDet-D0 improves by 2.41% compared with the original EfficientDet-D0 on the publicly available road damage dataset and outperforms other networks such as YOLOv5s, YOLOv7-tiny, YOLOv4-tiny, Faster R-CNN, and SSD. Meanwhile, the detection speed of EfficientDet-D0 can reach 27.0 FPS, which meets the demand for real-time detection, and the model size is only 32.31 MB, which is suitable for deployment in mobile devices such as unmanned inspection carts, UAVs, and smartphones. In addition, the detection accuracy of E-EfficientDet-D2 can reach 57.51%, which is 4.39% higher than E-EfficientDet-D0, and the model size is 61.78 MB, which is suitable for practical application scenarios that require higher detection accuracy and better hardware performance.