A system of four globally coupled doubling maps is studied in this paper. It is known that such systems have a unique absolutely continuous invariant measure (acim) for weak interaction, but the case of stronger coupling is still unexplored. As in the case of three coupled sites [14], we prove the existence of a critical value of the coupling parameter at which multiple acims appear. Our proof has several new ingredients in comparison to the one presented in [14]. We strongly exploit the symmetries of the dynamics in the course of the argument. This simplifies the computations considerably, and gives us a precise description of the geometry and symmetry properties of the arising asymmetric invariant sets. Some new phenomena are observed which are not present in the case of three sites. In particular, the asymmetric invariant sets arise in areas of the phase space which are transient for weaker coupling and a nontrivial symmetric invariant set emerges, shaped by an underlying centrally symmetric Lorenz map. We state some conjectures on further invariant sets, indicating that unlike the case of three sites, ergodicity breaks down in many steps, and not all of them are accompanied by symmetry breaking.