The role of language in mediating or augmenting human thought is the subject of long-standing debate. One specific claim links language and the ability to categorize objects based on a certain feature. According to this view, language resources are critical for feature-based categorization because verbal labels can help maintain focus on the relevant categorization criterion and reduce interference from other (irrelevant) features. As a result, language impairment is expected to affect categorization of items grouped according to a single feature (low-dimensional categories, e.g., ‘Things made of wood’), where many irrelevant features need to be inhibited, more than categorization of items that share many features (high-dimensional categories, e.g., ‘Animals’), where few irrelevant features need to be inhibited. We here present findings from individuals with aphasia that go against this hypothesis (Experiments 1 and 2). We also present fMRI data from young healthy adults, showing that the language brain regions exhibit low activity level during categorization, for both low-dimensional and high-dimensional categories (Experiment 3). In tandem, these results demonstrate that language is not critical for object categorization. Our work contributes to the growing evidence that, although language may assist in accessing task-relevant information (e.g., instructions), many cognitive tasks in adult brains proceed without recruiting the language system.