Polymer matrix composites (PMCs) with high thermal conductivity (TC) play an important role in improving the heat dissipation capacity of a new generation of electronic devices, particularly for 5G and aviation applications. Over the last few decades, considerable efforts have been made in the fabrication of highly thermally conductive PMCs. Advances in the thermal conduction mechanism of polymer composites are induced to, and then commonly used thermally conductive fillers are presented. In the following, the factors affecting the TC of polymer composites are discussed in detail, including fillers, interfaces, polymer matrices and processing technologies. Special attention is paid to the thermally conductive fillers. Then, some application areas of thermally conductive polymer composites are introduced. Finally, the deficiencies and future development trends in this research field are put forward. It is expected that this review will provide some beneficial inspiration in improving the TC of PMCs.