(1) Background: Growing evidence indicates that inflammation can induce neural circuit dysfunction and plays a vital role in the pathogenesis of major depressive disorder (MDD). Nevertheless, whether inflammation affects the integrity of white matter pathways is only beginning to be explored. (2) Methods: We computed quantitative anisotropy (QA) from diffusion magnetic resonance imaging as an index of white matter integrity and regressed QA on C-reactive protein (CRP), controlling for age, sex, and BMI, in 176 participants with MDD. (3) Results: The QA values of several white matter tracts were negatively correlated with CRP concentration (standardized beta coefficient = −0.22, 95%CI = −0.38–−0.06, FDR < 0.05). These tracts included the bilateral cortico-striatal tracts, thalamic radiations, inferior longitudinal fasciculi, corpus callosum (the forceps minor portion and the tapetum portion), cingulum bundles, and the left superior longitudinal fasciculus III. Importantly, the association remained robust after regressing up to twelve potential confounders. The bilateral fornix and a small portion of the thalamic radiation showed a positive association with CRP levels, but these associations did not remain significant after adjusting for confounders. (4) Conclusions: Peripheral inflammation may contribute to the etiology of MDD by impacting the microstructural integrity of brain corticolimbic white matter pathways.