As an ingredient of great economic value, Tricholoma matsutake has received widespread attention. However, heavy metal residues and preservatives in it will affect the quality of Tricholoma matsutake and endanger the health of consumers. Here, we present a method for the simultaneous detection of low concentrations of potassium sorbate and lead in Tricholoma matsutakes based on surface-enhanced Raman spectroscopy (SERS) and fluorescence (FLU) spectroscopy to test the safety of consumption. Data fusion strategies combined with multiple machine learning methods, including partial least-squares regression (PLSR), deep forest (DF) and convolutional neural networks (CNN) are used for model training. The results show that combined with reasonable band selection, the CNN prediction model based on decision-level fusion achieves the best performance, the correlation coefficients (R2) were increased to 0.9963 and 0.9934, and the root mean square errors (RMSE) were reduced to 0.0712 g·kg−1 and 0.0795 mg·kg−1, respectively. The method proposed in this paper accurately predicts preservatives and heavy metals remaining in Tricholoma matsutake and provides a reference for other food safety testing.