Hexakis(4-hydroxyphenoxy) cyclotriphosphazene (PN-OH) was synthesized by a nucleophilic substitution and aldehyde reduction reaction of hexachlorocyclotriphosphazene with p-hydroxybenzaldehyde, and then cyclophosphazene nucleated hyperbranched polyesters (HBP) were prepared via a one-pot esterification reaction of PN-OH and dimethylolpropionic acid. The chemical structure and molecular weight distribution of HBP were confirmed by 1 H-NMR, FTIR, and GPC. The HBP was applied to epoxy resin (EP) to improve its mechanical properties and flame retardancy. The cone calorimeter test revealed that the maximum heat release rate, total heat release and total smoke release of HBP/EP composites were reduced by 35.9%, 21.6%, and 27.3%, respectively, when 7 wt% HBP was added to the EP matrix compared with pure EP. Meanwhile, when 7 wt% HBP was added to EP composites, the limiting oxygen index value was increased from 24.6% to 27.7% and the UL-94 test reached a V-1 rating. More importantly, the thermal stability and mechanical properties of the HBP/EP composites were enhanced. The flame retardant mechanism analysis showed that HBP primarily acted as a flame retardant in the condensed phase. These results indicate that the HBP/EP composites have exceptional overall performance and a broad range of applications in critical fields.