This article examines how residents experience and account for stigma at Claremont Court, a modernist social housing scheme built in Edinburgh in the early 1960s. Although listed as having special architectural interest, the building has been subject to disinvestment and has a mix of residents, including council and private renters as well as owner-occupiers. This article explores micro-distinctions between residents, showing how the categories ‘stigmatiser’ and ‘stigmatised’ are not as rigid as we might expect. It then considers stigma associated not with residents but, rather, the building itself, and argues that closer attention to the relationship between the material and social is required in order to understand residents’ complex articulations of belonging. Finally, residents’ views on dirt and rubbish are explored, showing how they use these signifiers of stigma to reveal concerns about shame and respectability. Responding to the call from this journal for more sociological understandings of stigma, this article argues that interactionist approaches offer an important alternative, one that highlights how stigma is negotiated, resisted and apportioned in everyday life. This perspective reveals residents’ practices in interaction with the material environment, as well as the ways in which stigmatisation processes work simultaneously in upward and downward directions, rather than in a unidirectional way.