To extract weak signal from the chaotic background, in this paper we analyze the theory of state space reconstruction of complicated nonlinear system, and put forward an estimation method utilizing the least-squares support vector machine (LS-SVM) based on a generalized window function. In the algorithm the generalized embedded window is taken as a foundation and the correlation function method is used to determine the embedded dimension and time delay of Lorenz system and so the state space reconstruction is realized and by combining the error forecasting model in which the LS-SVM is used to estimate the errors, the detection of the weak target signal, such as transient and periodic signal, is achieved. It is illustrated in the simulation experiments that the model proposed can detect the weak signals effectively from a chaotic background and reduce the influence of noise on the target signals, which possesses minor forecasting error. Compared with those conventional methods, this method has a remarkable advantage in reducing detection threshold and improving the accuracy of prediction. When the signal-to-noise ratio is -87.41 dB in the chaotic noise background, the new method can reduce the root mean square error nearly two orders of magnitude, reach 0.000036123, while the traditional SVM can only reach 0.049 under the condition of -54.60 dB.