The etiology of prostate cancer (PCa) remains uncertain, and the role of diet is unclear. We aimed to evaluate the role of diet, through dietary patterns, on PCa, considering tumor aggressiveness and extension. The CAPLIFE study is a population-based case-control study including a total of 428 incident PCa cases and 393 controls aged 40–80 years. Dietary information was collected through a validated food frequency questionnaire. Three dietary patterns were identified through principal component analysis: “Mediterranean,” “Western,” and “Unhealthy,” which were categorized into tertiles according to the control group cutoff points. Tumor aggressiveness and extension was determined. Logistic regression models were used to assess the association between dietary patterns and PCa. High adherence to an unhealthy dietary pattern was associated with higher odds of PCa, ORT3vsT1 = 1.52 (95% CI 1.02–2.27), especially for cases with ISUP 1–2 and localized PCa tumors. This association was not observed with a Western or Mediterranean pattern. In conclusion, adherence to an unhealthy diet appears to be associated with higher odds of PCa, especially for cases with ISUP 1–2 and localized PCa tumors.