Anyons in a topologically ordered phase can carry fractional quantum numbers with respect to the symmetry group of the considered system, one example being the fractional charge of the quasiparticles and quasiholes in the fractional quantum Hall effect. When such symmetry-fractionalized anyons condense, the resulting phase must spontaneously break the symmetry and display a local order parameter. In this paper, we study the phase diagram and anyon condensation transitions of a Z 2 topological order perturbed by Ising interactions in the Toric Code. The interplay between the global ("onsite") Ising (Z 2 ) symmetry and the lattice space group symmetries results in a non-trivial symmetry fractionalization class for the anyons, and is shown to lead to two characteristically different confined, symmetry-broken phases. To understand the anyon condensation transitions, we use the recently introduced critical torus energy spectrum technique to identify a line of emergent 2+1D XY* transitions ending at a fine-tuned ( Ising2 )* critical point. We provide numerical evidence for the occurrence of two symmetry breaking patterns predicted by the specific symmetry fractionalization class of the condensed anyons in the explored phase diagram. In combination with large-scale quantum Monte Carlo simulations we measure unusually large critical exponents η for the scaling of the correlation function at the continuous anyon condensation transitions, and we further identify lines of (weakly) first order transitions in the phase diagram. As an important additional result, we discuss the phase diagram of a resulting 2+1D Ashkin-Teller model, where we demonstrate that torus spectroscopy is capable of identifying emergent XY/O(2) critical behaviour, thereby solving some longstanding open questions in the domain of the 3D Ashkin-Teller models. To establish the generality of our results, we propose a field theoretical description capturing the transition from a Z 2 topological order to either Z 2 symmetry broken phase, which is in excellent agreement with the numerical results.