Current cognitive models suggest that the processing of dynamic facial attributes, including social signals such as gaze direction and facial expression, involves the superior temporal sulcus, whereas the processing of invariant facial structure such as the individuals' identity involves the fusiform face area. Where facial attractiveness, a social signal that may emerge from invariant facial structure, is processed within this dual-route model of face perception is uncertain. Here, we present two studies. First, we investigated the explicit judgments of facial attractiveness and attractiveness-motivated behaviour in patients with acquired prosopagnosia, a deficit in familiar face recognition usually associated with damage to medial occipitotemporal cortex. We found that both abilities were impaired in these patients, with some weak residual ability for attractiveness judgments found only in those patients with unilateral right occipitotemporal or bilateral anterior temporal lesions. Importantly, deficits in attractiveness perception correlated with the severity of the face recognition deficit. Second, we performed a functional magnetic resonance imaging study in healthy subjects that included an implicit and explicit processing of facial attractiveness. We found increased neural activity when explicitly judging facial attractiveness within a number of cortical regions including the fusiform face area, but not the superior temporal sulcus, indicating a potential contribution of the fusiform face area to this judgment. Thus, converging neuropsychological and neuroimaging evidence points to a critical role of the inferior occipitotemporal cortex in the processing of facial attractiveness.