The paper presents the results of research concerning the influence of micromaterials on the heat conductivity coefficient λ, specifically heat Cp and thermal diffusivity a of modified gypsum and geopolymer. Microspheres, hydroxyethyl methylcellulose (HEMC) polymer, and aerogel were used as the gypsum’s modifying materials. The study also investigated an alkali potassium-activated methakaolin-based geopolymer with the addition of aluminium dust. During the measurements of thermal parameters, the nonstationary method was chosen, and an Isomet device—which recorded the required physical quantities—was used. When compared to the reference sample, a decrease in the thermal conductivity and diffusivity of the hardened gypsum— and a simultaneous increase in specific heat—was observed with the addition of micromaterials. The geopolymer sample was characterized by the lowest value of thermal conductivity, equal to 0.1141 W/(m·K). It was over 62% lower than the reference sample containing only gypsum. The experimental values of the thermal conductivity of the gypsum samples with the addition of HEMC, aerogel and microspheres were, respectively, over 23%, 6%, and 8% lower than those of the unmodified gypsum samples. The lowest values of thermal conductivity were observed in the case of the gypsum samples modified with polymer; this resulted from the fact that the polymer caused the greatest change in the structure of the gypsum’s composite, which were expressed by the lowest density and highest porosity.