Making attention mechanisms more robust and interpretable with virtual adversarial training

Multimodal Anomaly Detection for High-Speed Train Control System Based on Attention Mechanism

“…The calculation of the attention mechanism (Kitada et al 2023) is split into two steps: rstly, the attention distribution is computed over all inputs X, and then, based on the result, the weighted average of the input information is calculated.…”