Zirconium boride (ZrB2) with high electrical, thermal conductivity, chemical stability and low coefficient of thermal expansion has been considered promising boride ceramic. In this study, the new type polyimide(PI)-based composites filled with ZrB2 nanopellets were prepared via in-situ polymerization. The results show that the ZrB2 nanopellets are dispersed uniformly in the PI matrix and the ZrB2/PI composites with 0.5 wt.% loading exhibits outstanding dielectric, mechanical properties and thermal stability. A mechanism related to the microstructure is proposed to explain the performance improvement. ZrB2 with strong electron affinity could trigger charges to accumulate on its surface, which promotes the interface polarization and consequently enhance dielectric constant. The strong interactions between ZrB2 and PI beneficial to stress effectively transfer from the PI matrix to the ZrB2. The dense structure of the composites and the intrinsically high thermal conductivity of ZrB2 are conducive to heat transfer in the matrix and reduce heat accumulation. The dielectric constant, tensile strength and elongation at break of the composites are increased by 17.5%, 63% and 59.4% compared to pure PI. Meanwhile, the ZrB2/PI composites possess superior electrical insulation property. The comprehensive performance improvement of the composites share a broader prospect for its application in engineering.