Listening-related fatigue is a potentially serious negative consequence of an aging auditory and cognitive system. However, the impact of age on listening-related fatigue, and the factors underpinning any such effect, remain unexplored. Using data from a large sample of adults (N = 281), we conducted a conditional process analysis to examine potential mediators and moderators of age-related changes in listening-related fatigue. Mediation analyses revealed opposing effects of age on listening-related fatigue; aging was associated with increased listening-related fatigue for individuals with higher self-reported hearing impairment, but also decreased listening-related fatigue via reductions in mood disturbance and sensory processing sensitivity (‘sensitivity’). Results also suggested that the effect of auditory attention ability on listening-related fatigue was moderated by sensitivity; for individuals with high sensitivity, better auditory attention ability was associated with increased fatigue. These findings shed light on the perceptual, cognitive, and psychological factors underlying age-related changes in listening-related fatigue.