Tea polyphenols are important ingredients for evaluating tea quality. The rapid development of sensors provides an efficient method for nondestructive detection of tea polyphenols. Previous studies have shown that features obtained from single or multiple sensors yield better results in detecting interior tea quality. However, due to their lack of external features, it is difficult to meet the general evaluation model for the quality of the interior and exterior of tea. In addition, some features do not fully reflect the sensor signals of tea for several categories. Therefore, a feature fusion method based on time and frequency domains from electronic nose (E-nose) and hyperspectral imagery (HSI) is proposed to estimate the polyphenol content of tea for cross-category evaluation. The random forest and the gradient boosting decision tree (GBDT) are used to evaluate the feature importance to obtain the optimized features. Three models based on different features for cross-category tea (black tea, green tea, and yellow tea) were compared, including grid support vector regression (Grid-SVR), random forest (RF), and extreme gradient boosting (XGBoost). The results show that the accuracy of fusion features based on the time and frequency domain from the electronic nose and hyperspectral image system is higher than that of the features from single sensor. Whether based on all original features or optimized features, the performance of XGBoost is the best among the three regression algorithms (R2 = 0.998, RMSE = 0.434). Results indicate that the proposed method in this study can improve the estimation accuracy of tea polyphenol content for cross-category evaluation, which provides a technical basis for predicting other components of tea.