Anthracnose is the most important disease of olives responsible for olive oil depreciation. The aim of this study is to compare the negative effects of the most common olive anthracnose pathogens in Portugal, Colletotrichum nymphaeae, C. godetiae, or C. acutatum, on oil extracted from fruits of the most cultivated Portuguese olive cultivars, ‘Galega Vulgar’ and ‘Cobrançosa’. Healthy fruits (ripening index 3.5) were inoculated, incubated, and sampled over 14 days post-inoculation (d.p.i.) for symptom assessment and oil extraction. Virgin olive oil (VOO) was characterised by quality criteria parameters (acidity, peroxide value, UV absorbances, organoleptic assessment, fatty acid ethyl esters (FAEE)), total phenols (TPH), main fatty acids, and waxes compositions. Galega Vulgar fruits were more susceptible to anthracnose than Cobrançosa fruits. Colletotrichum nymphaeae was equally highly virulent for both cultivars, while C. godetiae was less virulent especially in Cobrançosa. VOO acidity increased reached 5% in Galega oils and 2% in Cobrançosa oils from fruits infected with C. nymphaeae or C. acutatum at 14 d.p.i. FAEE were higher than the legal limit for extra VOO in Galega oils from fruits at 7 d.p.i. with C. nymphaeae or C. acutatum, or 11 d.p.i. with C. godetiae. TPH decreased through the experiments with all the inoculated fungi. The musty sensory defect was detected in Galega and Cobrançosa VOO from fruits inoculated with C. nymphaeae or C. acutatum at 3 or 7 d.p.i. earlier than chemical degradation, respectively. The degradation levels of olive oils depended on the Colletotrichum species and olive cultivar.