A confluence of societal changes, particularly hardening racial attitudes following the Indian Mutiny in 1857 and the Morant Bay Rebellion in 1865, resulted in widescale disillusionment with imperial humanitarian projects in the middle decades of the nineteenth century. As this article demonstrates, however, the membership and income of the Aborigines’ Protection Society (APS) increased at precisely the moments when this disillusionment was at its sharpest. This article combines quantitative and qualitative methods to assess the nature of the Society's mid-century membership base, demonstrating that, rather than a monolithic decline, a humanitarian polarization took place in response to imperial crises that led some (largely Tories) to disillusionment and others (largely Whigs) to entrenchment. Furthermore, by attending to discursive trends within speeches at APS annual meetings as well as in private correspondence between members and the secretary of the Society, I explore how APS members explained the connection between their own lives and the treatment of distant Indigenous peoples in the colonies. Finding that British Indigenous rights activism was only seldomly expressed in terms of Indigenous peoples themselves, I show that support for the APS was most commonly related to concerns for friends and family living in the colonies, along with disquiet about the impact of colonial injustices on international competition. This enabled Indigenous rights activists to continue their efforts in the face of disillusionment with the capabilities of racialized “others.”