The influence of locally different species interactions on trait evolution is a focus of recent evolutionary studies. However, few studies have demonstrated that geographically different pollinator-mediated selection influences geographic variation in floral traits, especially across a narrow geographic range. Here, we hypothesized that floral size variation in the Japanese herb Prunella vulgaris L. (Lamiaceae) is affected by geographically different pollinator sizes reflecting different pollinator assemblages. To evaluate this hypothesis, we posed two questions. (1) Is there a positive correlation between floral length and the proboscis length of pollinators (bumblebees) across altitude in a mountain range? (2) Does the flower-pollinator size match influence female and male plant fitness? We found geographic variation in the assemblage of pollinators of P. vulgaris along an altitudinal gradient, and, as a consequence, the mean pollinator proboscis length also changed altitudinally. The floral corolla length of P. vulgaris also varied along an altitudinal gradient, and this variation strongly correlated with the local pollinator size but did not correlate with altitude itself. Furthermore, we found that the size match between the floral corolla length and bee proboscis length affected female and male plant fitness and the optimal size match (associated with peak fitness) was similar for the female and male fitness. Collectively, these results suggest that pollinator-mediated selection influences spatial variation in the size of P. vulgaris flowers at a fine spatial scale.