The topologically protected vortex–antivortex (V–AV) domain structure in ferroelectric hexagonal manganites has been highly concerned recently, but its stability against intrinsic defects remains to be understood, given the claim that a topological structure would be robust against defects and other perturbations. In fact, it is also known that the V–AV structure is sensitive to the sample quality, and such a well-developed structure is hardly observed in thin films and defective single crystals. In this work, we investigate the influence of anti-trimer point defects on the stability of the V–AV domain structure by employing the phase-field simulation based on the Landau–Devonshire phenomenological theory. It is revealed that the characteristic V–AV structure essentially relies on the anti-trimer point defects under consideration. These defects lower the trimerization transition temperature on one hand and produce pinning effect on the vortex cores/walls on the other hand. However, the V–AV structure does remain robust if the anti-trimer magnitude of these defects is relatively weak but will be eventually destroyed if the anti-trimer magnitude is strong.