The problem of insignificant image features and poor image quality in the acquisition of terahertz spectral images of unsound wheat. Therefore, this study proposed the FFDNet-VGG terahertz image enhancement algorithm based on the FFDNet denoising algorithm and VGG feature extraction algorithm. The experimental results showed that this algorithm could effectively improve the image quality compared with the traditional algorithms such as BM3D and WNNM, and the PSNR and SSIM of the enhanced images were 39.10 dB and 0.93, respectively. The wheat unsound grain images processed by FFDNet, VGG and FFDNet-VGG algorithms were verified by CNN classification network with the classification accuracy of 93.1% and 93.7%, respectively. FFDNet-VGG effectively enhances the wheat unsound grain terahertz spectral images.