Background: Breast cancer subtypes are known to have different sites of metastatic recurrence. Distant metastases are often seen during the post-operative course in patients with human epidermal growth factor receptor 2 (HER2)-enriched breast cancer (HER2BC) and triple-negative breast cancer (TNBC) while being relatively rare in those with hormone receptor-positive and HER2-negative breast cancer (HR+HER2-BC). Tumor-infiltrating lymphocytes (TILs) can serve as an index to monitor tumor immune microenvironment and possibly predict the prognosis and therapeutic effect in breast cancer. This study aimed to investigate the correlation between TIL density and recurrence site in HR+HER2-BC.Methods: Four-hundred and seventy-one patients with HR+HER2-BC underwent surgery as the first treatment and received adjuvant endocrine therapy except adjuvant chemotherapy at the Osaka City University Hospital from April 2007 to October 2015. To evaluate tumor morphology and examine TILs, needle biopsy specimens were used. Morphological assessment was conducted using conventional hematoxylin and eosin staining.Results: Forty-two patients had a recurrence of breast cancer. In patients with no TIL density, local recurrence was significantly less (p = 0.022), while distant metastases were significantly more (p = 0.015) compared to those in patients with TIL density. Therefore, for the prediction of distant metastases in HR+HER2-BC without chemotherapy, TILs could be used as predictors in univariate analysis (p = 0.015, odds ratio [OR] = 0.127), although not as independent factors (p = 0.285, OR = 0.144).Conclusions: We concluded that TILs may be able to predict distant metastatic recurrence in stages I–II of HR+HER2-BC.