Stigma by association is described in qualitative research of family members who have relatives diagnosed with mental illness, depicting their sense of public shame for having these relationship ties. However, there have been relatively few empirical studies thus far, in part due to the isolation of family members affecting research recruitment. In order to address this gap, an online survey was administered to 124 family members, comparing those who live in the same home with their ill relative (n = 81) and those who do not (n = 43). A remarkable incidence of one in three family members reported experiencing stigma by association. Those living with an ill relative reported comparatively higher levels of stigma by association using an adapted questionnaire measure. Both groups experienced loneliness (moderate levels), but importantly, the cohabiting relatives perceived themselves as lacking support from friends and other family members. Correlational analyses revealed that those with heightened stigma by association reported heightened anti-mattering: that is, feeling that other people treat them as if they are insignificant and invisible. Anti-mattering was also associated with more loneliness and reduced social support. Our discussion focuses on the theme that family members who actually live with mentally ill relatives experience heightened social isolation that is under-recognized due to public stigma concerns, compounded by feeling their own lives do not matter to others. Public health implications are considered for the stigmatized family members who appear to be particularly marginalized.