The total phenols in virgin olive oil are highly dependent on cultivar, but also on ripening stage and other agronomic factors. The focus of most studies on agronomic factors has been irrigation, while fertilization has received less attention. Most of the fertilization works find that nitrogen over-fertilization leads to a decrease in phenol contents in virgin olive oil (VOO) and extra virgin olive oil (EVOO), under rain-fed or irrigation management. Ortho-diphenols also decrease with high doses of nitrogen, with no effect on secoiridoids. Phosphorous has a minor effect on irrigated trees; while the role of potassium is controversial, with a lack of trials with calcium and micro-nutrients. Due to the great impact of the fertilization on the phenol content and quality of VOO, new research is necessary with focus aimed at different cultivars and agronomic factors.