The tumor microenvironment (TME) in breast cancer comprises local factors, cancer cells, immune cells and stromal cells of the local and distant tissues. The interaction between cancer cells and their microenvironment plays important roles in tumor proliferation, propagation and response to therapies. There is increasing research in exploring and manipulating the non-cancerous components of the TME for breast cancer treatment. As the TME is now increasingly recognized as a treatment target, its pathologic assessment has become a critical component of breast cancer management. The latest WHO classification of tumors of the breast listed stromal response pattern/fibrotic focus as a prognostic factor and includes recommendations on the assessment of tumor infiltrating lymphocytes and PD-1/PD-L1 expression, with therapeutic implications. This review dissects the TME of breast cancer, describes pathologic assessment relevant for prognostication and treatment decision, and details therapeutic options that interacts with and/or exploits the TME in breast cancer.