The Impact Echo (IE) method is a widely used method for the detection of concrete structures. The signal has non-stationary characteristics. Empirical mode decomposition (EMD) is one of the typical ways of dealing with non-stationary signals. Due to EMD having mode overlap and a breakpoint effect, Variational Mode Decomposition (VMD) has been proposed to overcome these problems. In this study, VMD is introduced into the analysis of the concrete component detection signals, and the impact echo analysis method based on VMD and marginal spectrum is established. The proposed method to solve the marginal spectrum is used in order to analyze the numerical simulation model's IE signal. This method is able to correctly obtain the main frequency. When compared with the traditional Fourier spectrum, the marginal spectrum that is based on the VMD method shows that the band width that is present in the main frequency is narrow. Even in the presence of a highly noisy background, the method is able to correctly obtain the main frequency. At last, through the detection signal analysis of the non-defective concrete specimen and the concrete specimen with the circular hole, the results show that the proposed method can distinguish the signal in the different defective states of the concrete structures, which has obvious advantages compared with the Fourier spectrum.