Due to the variety of materials' forms and shapes as well as the dearth of material annotation data, the work of semantic segmentation presents considerable difficulties with material segmentation. Additionally, traditional RGB images are insufficient for obtaining adequate segmentation results. A polarization-driven material semantic segmentation method combined with a self-attention mechanism is proposed as a solution to these problems. The backborn network ResNet50, the decoding head DeepLabV3+, and the auxiliary decoding head FCN make up a unique network architecture. For the decoder and encoder in the decoding, and auxiliary decoding head, the polarization self-attention modules are introduced in order to preserve high-quality internal resolution in both the spatial and channel dimensions. Rich texture and edge information are provided by the polarization information, which is used as auxiliary information. Furthermore, there is a problem with the class imbalance in the application dataset. On the MCubeS dataset, comparative and ablation experiments are carried out. The outcomes show that the suggested method outperforms baseline methods by 7.92 percentage points. Furthermore, the proposed strategy improves by 1.15 percentage points in comparison to the network without a polarization self-attention mechanism.