Terahertz (THz) nondestructive testing has emerged as a promising tool for public security check due to its superior detection accuracy and sensitivity for hidden objects under clothing. However, it is unknown if material composition and textile structure of clothing will influence the THz spectrum of the hidden objects. In this paper, the THz time domain spectroscopy (THz-TDS) was used to detect several natural and artificial fabrics containing pure amino acids with characteristic absorption peaks between 0.2 and 1.6 THz. The results show that the absorption peaks of AAs of various cloth materials have different degrees of decrease and redshift. In addition, the absorption peaks of the THz spectrum disappear when the textile structure is destroyed. Moreover, the simulation results also verify that the textile structure has an obvious effect on the THz field distribution. In this sense, it is necessary to analyze the material composition and textile structure when detecting hidden objects under clothing using the THz signal.