We review research on the neural correlates of colour categories, and categorical responses in preverbal infants and animals. Our aim was to address the question of the functional segregation of colour categories from colour perception and colour language, i.e. whether or not colour categorisation shares processing resources with language and perception. Together, the reviewed findings suggest that colour categorisation often involves automatic language processing, but it can also occur in the absence of language or colour naming, as suggested by research on brain-damaged patients, preverbal infants, and non-human animals. Furthermore, there is no compelling evidence coming from human neuroimaging and neuropsychological studies, or from animal studies, that categorisation is inherent to colour perception. Instead, colour categorisation might simply build upon the continuous perception of colour and may interact with perception through the direction of attention to colour differences that are relevant to categorisation. Altogether, the reviewed evidence does not seem to offer a straightforward account of the origin of lexical colour categories. We suggest that future research in all areas (1) requires methodological improvements, in particular in stimulus control, (2) should overcome the universalist-realist debate and go beyond a simple contrast between perception and language, and (3) should consider the link between object colours and colour categories to establish both evolutionary and developmental continuity between categorical responses in animals, infants and adult humans.