Structural damage detection is crucial for ensuring the safety and reliability of civil infrastructure. In this paper, we propose a novel vibration-based structural damage detection framework using Spectral Distance and a combination of t-Distributed Stochastic Neighbor Embedding (t-SNE) and Gaussian Mixture Model (GMM). By integrating these three procedures, a unique, comprehensive and highly effective approach for detecting structural damage is achieved. First, we extract spectral distance-based features from structural response data under ambient excitations, which provide information about the deviation of the spectral content of the signal from the undamaged structural response. Then, we apply t-SNE to embed the dimensional data into lower dimensional feature space and visualize the data in a low-dimensional space. Finally, we use GMM-based clustering to classify the data into normal and anomalous classes. To evaluate the proposed approach, the experimental data from the Phase II benchmark structural health monitoring problem introduced by the IASC-ASCE Structural Health Monitoring Task Group, is utilized. The results show that our method can effectively detect structural damage anomalies. Moreover, the proposed framework is computationally efficient, making it suitable for real-time structural health monitoring applications. This paper presents novel structural damage indicators (SDI) based on power spectral density deviations between damaged and undamaged structural responses. The study compares the performance of proposed spectral-distance measures and emphasizes the significance of spectral Kullback-Leibler distance as an indicator of structural deterioration. Additionally, the effectiveness of t-SNE-GMM is demonstrated in efficiently segregating damage configurations.