Thermal interface materials (also known as thermal pads) are widely used as a crucial part to dissipate heat generated in miniaturized and integrated electronic components. Here, we systematically investigated the effects of small ceramic and metallic powders in rubbery thermal composite pads with a high content of aluminum oxide filler on the thermal conductivity of the composite pads. We optimized the compositions of aluminum oxide fillers with two different sizes in a polydimethylsiloxane (PDMS) matrix for rubbery composite pads with a high thermal conductivity. Based on the optimized compositions, zinc oxide powder or copper powder with an average size of 1 μm was used to replace 5 μm-sized aluminum oxide filler to examine the effects of the small ceramic and metallic powders, respectively, on the thermal conductivity of the composite pads. When zinc oxide powder was used as the replacement, the thermal conductivity of the rubbery composite pads decreased because more air bubbles were generated during the processing of the mixed paste with increased viscosity. On the other hand, when the copper powder was used as a replacement, a thermal conductivity of up to 2.466 W/m·K was achieved for the rubbery composite pads by optimizing the mixing composition. SEM images and EDS mapping confirmed that all fillers were evenly distributed in the rubbery composite pads.