Categorising gender for soft biometric recognition is especially challenging from low quality surveillance footage. Our novel approach discovers super fine-grained visual taxonomies of gender from pairwise similarity comparisons, annotated via crowdsourcing. This paper presents our techniques for collection, interpretation and clustering of perceived visual similarities, and discusses the transition from pre-defined categorisation to similarity comparisons between subjects. We compare and evaluate our proposal on two diverse datasets, demonstrating the ability to describe multiple concepts, including ambiguity and uncertainty, that go beyond binary male-female designators. Our method is applicable to a wide range of soft biometric traits and image attributes, and can aid in efficiently annotating large-scale datasets, by generating more discriminative, reproducible and flexible categorical labels.