The widespread application of audio and video communication technologies has resulted in compressed audio data flowing across the Internet and has made it an important carrier for covert communication. Many steganographic schemes have emerged for mainstream audio data compression, including AAC and MP3, with many steganalysis schemes being subsequently developed. However, these steganalysis schemes are only effective in the given embedding domain. In this paper, a general steganalysis scheme, Spec-ResNet (Deep Residual Network of Spectrogram), is proposed to detect steganography schemes in different embedding domains for AAC and MP3. The basic idea is that the steganographic modification of different embedding domains all introduce changes in the decoded audio signal. In this paper, a spectrogram, which is a visual representation of the spectrum of the frequencies of an audio signal, is adopted as the input of the feature network to extract the universal features introduced by steganography schemes. Our deep-neural-networkbased method, Spec-ResNet, can effectively represent steganalysis features, and the features extracted from different spectrogram windows are combined to fully capture the steganalysis features. The experimental results show that the proposed scheme achieves good detection accuracy and generality. The proposed scheme achieves higher detection accuracy for three different AAC steganographic schemes and MP3Stego compared to state-ofthe-art steganalysis schemes based on traditional hand-crafted or CNN-based features. To the best of our knowledge, this audio steganalysis scheme based on spectrograms and a deep residual network is the first to be published. The method proposed in this paper can be extended to the audio steganalysis of other codecs or audio forensics.