To what extent is the size of the blood-oxygen-level-dependent (BOLD) response influenced by factors other than neural activity? In a re-analysis of three neuroimaging datasets, we find large systematic inhomogeneities in the BOLD response magnitude in primary visual cortex (V1): stimulus-evoked BOLD responses, expressed in units of percent signal change, are up to 50% larger along the representation of the horizontal meridian than the vertical meridian. To assess whether this surprising effect can be interpreted as differences in local neural activity, we quantified several factors that potentially contribute to the size of the BOLD response. We find strong relationships between BOLD response magnitude and cortical thickness, cortical curvature, and the presence of large veins. These relationships are consistently found across subjects and suggest that variation in BOLD response magnitudes across cortical locations reflects, in part, differences in anatomy and vascularization. To compensate for these factors, we implement a regression-based correction method and show that after correction, BOLD responses become more homogeneous across V1. The correction reduces the horizontal/vertical difference by about half, indicating that some of the difference is likely not due to neural activity differences. Additionally, we find that while the cerebral sinuses overlap with the vertical meridian representation in V1, they do not explain the observed horizontal/vertical difference. We conclude that interpretation of variation in BOLD response magnitude across cortical locations should consider the influence of the potential confounding factors of cortical thickness, curvature, and vascularization.