The structure of a gypsum matrix, which is characterized by the size, shape, and morphology of crystals, the state of the interfacial surface, and the number and strength of contacts between new growths, determines the basic physical and technical properties of materials. The effect of various modifiers, mainly of technogenic origin, on the gypsum stone structure has been examined by scanning electron microscopy and energy-dispersive X-ray analysis. In the study, building gypsum and metallurgical dust, granulated blast furnace slag, and siltstone mineral modifiers have been used. It has been established that the introduction of modifiers improves the physical and mechanical properties of a binder. This is probably due to the fact that the additives act as crystallization centers during the matrix structure formation and facilitate filling of the stone pore space.