Polymer-based dielectrics with high energy storage density are attracting increasing attention due to their wide applications in pulsed-discharge and power conditioning electronic fields. Despite some numerical simulation about effects of horizontally arranged and vertically arranged fibers on dielectric properties of composites already studied, the influence mechanism of the specific orientation and aspect ratio still remains to be studied. In this work, the effects of orientation angles and aspect ratios of nanofiber fillers on breakdown behavior and dielectric properties of composites are theoretically analyzed by the finite element and phase-field method. The results show that the more inclined the nanofiber fillers is, the higher the nominal breakdown strength is, which benefits from the obstruction of the conductive channels by the nanofibers. However, the dielectric constant shows the opposite law, which is the result of the decreased polarization along the electric field direction of the nanofiber fillers. Besides this, by modulating the distribution of local electric field, a higher aspect ratio of nanofiber fillers helps to achieve a much higher breakdown strength with a slight sacrifice of dielectric constant. The present work provides a comprehensive and quantitative understanding of the orientation and aspect ratio of nanofiber fillers on the dielectric and breakdown properties of composites, providing important guidance on optimizing the energy storage performance.