In the realm of structural health monitoring (SHM), understanding the expected behavior of a structure is vital for the timely identification of anomalous activities. Existing methods often model only the physical quantities of monitoring data, neglecting the corresponding temporal information. To address this, this paper presents an innovative deep learning framework that synergistically combines a BiLSTM model, fortified by a temporal pattern attention (TPA) mechanism, with time-encoded temperature and traffic-induced deflection-temporal patterns. The arithmetic optimization algorithm (AOA) is employed for optimal hyperparameter tuning, and incremental learning was implemented to enable real-time updates of the model. Based on the proposed framework, an anomaly detection method was subsequently developed. This method is bidirectional: it uses quantile loss to provide expected ranges for structural behavior, identifying isolated anomalies, while the windowed normalized mutual information (WNMI) based on multivariate kernel density estimation (MKDE) helps detect trend variability caused by decreases in structural stiffness. This framework and the anomaly detection method were validated using data from an operational cable-stayed bridge. The results demonstrate that the method effectively predicts structural behavior and detects anomalies, highlighting the critical role of temporal information in SHM.