The vulnerability of buildings impacted by flash floods is often assessed by empirical approaches. However, the damage-developing mechanisms of buildings remain unclear. This study presented a nonlinear numerical analysis for the interaction between flash floods and masonry structures. A new Structured ALE (S-ALE) solver in an explicit finite element platform LS-DYNA was applied. Thirty different scenarios, including two different FE models, three different water depths, and five different flow velocities, were studied. Nonlinear structural responses and failure mechanisms of masonry structures were analyzed and compared. Results showed that due to the consideration of wall damage, the time evolution of the impact force on the entire building was distributed in a multipeak pattern. Under the impact of flash floods, the building was in a complex bending-shear state, and the overturning moment was the principal reason for the building damage. The critical role played by the structural measure was reaffirmed. Moreover, the physical vulnerability was quantified through a macroscopic damage index, the lateral drift ratio of the ground floor. It can be concluded that the physical vulnerability depends on both the local structural strength (local view) and the structural resistance hierarchies (global view).