The attempts to improve the security of the speech signals are increasing with new encryption schemes. Many recent researches of speech encryption algorithms have been increasingly based on chaotic systems, but the drawbacks of weak security and small key space in one-dimensional chaotic cryptosystems are conspicuous. In this article, a new speech encryption scheme is used. It employs one of the three-dimensional chaotic systems to encrypt a source signal by permutations, thereby confusing the relationship between the cipher signal and the plain signal. A higher-order polynomial equation and rotating matrix are used to strengthen the proposed method. It significantly increases the resistance to attacks. The proposed system has the advantages of smaller iteration times and bigger key space. The system has entertained high-security analysis such as the signal-to-noise ratio, peak signal-to-noise ratio, auto correlation, key sensitivity analysis, number of sample change rate, unified average changing intensity, and histogram analysis. The analysis results illustrate that the proposed system is highly efficient and robust to threats.