Recent and past retreat of marine-terminating glaciers are broadly consistent with observed ocean warming, yet responses vary significantly within regions experiencing similar ocean conditions. We assess how fjord geometry modulates glacier response to a regional ocean warming on decadal to millennial time scales, by using an idealized, numerical model of fast-flowing glaciers including a crevasse-depth calving criterion. Our simulations show that, given identical climate forcing, grounding line responses can differ by tens of kilometers due to variations in channel width. We identify fjord mouths and embayments as vulnerable geometries, showing that glaciers in these fjords are prone to rapid, irreversible retreat, independent of the presence of a fjord sill. This irreversible retreat has relevance for the potential future recovery of marine ice sheets, if the current anthropogenic warming is reduced, or reversed, as well as for the response of marine ice sheets to past climate states; including the warm Bølling-Allerød interstadial, the Younger Dryas cold reversal and the Little Ice Age.