The study of meaning is a cornerstone of sociology, but the context-dependence of meaning is also a challenge for sociologists, especially sociologists of race. This article offers the Sociology of the Symbolic Landscape (SSL) as the theoretical solution to the challenge of studying meaning and applying the concept of race within and across contexts. SSL focuses on the relationships between symbolic categories within a context and their positions in the core/periphery distinction. Incorporating insights from Durkheim, Interpretive Sociology, the study of Symbolic Boundaries, Civil Sphere Theory, and Postcolonial Theory, SSL makes it possible to identify and categorize positions, thereby identifying the way meaning acts as a source of constraint and of creative possibilities, and enabling cross-contextual comparison of culture-structures and forms. Through application of SSL to data on categorizations in Sweden, we demonstrate the analytical traction SSL provides. SSL offers a portable understanding of the racial other as “outsider within,” a distinct position of enduring alterity and persistent difference within the symbolic landscape. Unlike outward-looking categories defined on the basis of a pre-existing religious attachment, ethnic group, or non-native nationality, the outsider within is an internally referenced category defined in negation – social membership with problematic and atypical belonging.