Damage caused by freezing wet, porous materials is a widespread problem, but is hard to predict or control. Here, we show that polycrystallinity makes a great difference to the stress build-up process that underpins this damage. Unfrozen water in grain-boundary grooves feeds ice growth at temperatures below the freezing temperature, leading to the fast build-up of localized stresses. The process is very variable, which we ascribe to local differences in ice-grain orientation, and to the surprising mobility of many grooves -which further accelerates stress build-up. Our work will help understand how freezing damage occurs, and in developing accurate models and effective damage-mitigation strategies.