Unmanned aerial vehicle (UAV)-assisted communication based on automatic modulation classification (AMC) technology is considered an effective solution to improve the transmission efficiency of wireless communication systems, as it can adaptively select the most suitable modulation method according to the current communication environment. However, many existing deep learning (DL)-based AMC methods cannot be directly applied to UAV platform with limited computing power and storage space, because of the contradiction between accuracy and efficiency. This paper mainly studies the lightweight of DL-based AMC networks to improve adaptability in resource-constrained scenarios. To address this challenge, we propose an ultra-lightweight neural network (ULNN). This network incorporates a lightweight convolutional structure, attention mechanism, and cross-channel feature fusion technique. Additionally, we introduce data augmentation (DA) based on signal phase offsets during the model training process, aimed at improving the model’s generalization ability and preventing overfitting. Through experimental validation on the public dataset RML2016.10 A, the ULNN we proposed achieves an average precision of 62.83% with only 8815 parameters and reaches a peak classification accuracy of 92.11% at SNR = 10 dB. The experimental results show that ULNN can achieve high recognition accuracy while keeping the model lightweight, and is suitable for UAV platform with limited resources.