The aim of this cross-sectional study was to assess the factors associated with the impact of oral health on the quality of life in a sample of 504 Brazilian independent elderly. Data collection included oral examinations and structured interviews. The simplified form of the Oral Health Impact Profile (OHIP-14) was used to measure OHRQoL. Information on sociodemographic characteristics, use of dental services, and subjective measures of health was collected. Poisson regression within a hierarchical model was used to data analyses. The following variables were associated with a negative impact on OHRQoL: female gender (PR = 1.40; CI 95%: 1.11–1.77); lower class (PR = 1.58; CI 95%: 1.13–2.20); up to 3 occluding pairs of posterior teeth (PR = 1.88; CI 95%: 1.13–3.14); at least one untreated caries (PR = 1.28; CI 95%: 1.06–1.54); curative reasons for the last dental appointment (PR = 1.52; CI 95%: 1.15–2.00); poor self-perception of oral health (PR = 2.49; CI 95%: 1.92–3.24); and poor perception of dental care provided (PR = 1.34; CI 95%: 1.12–1.59). The younger elderly also noticed this negative impact. These findings showed that the clinical, sociodemographic, and subjective factors evaluated exerted a negative impact on OHRQoL in elderly people. Health authorities must address all these factors when planning interventions on oral health for this population.