Research on the effects of social media on wellbeing has been equivocal. Social media impacts people’s wellbeing indifferent ways, but relatively little is known about why this is the case. Here we introduce the construct of “Social Media Sensitivity” to understand how social media’s effects on wellbeing differs across people and the contexts in which these platforms are used. In a month-long large-scale intensive longitudinal study (total n = 1,632; total number of observations = 120,599), we examined for whom and under which circumstances social media was associated with positive and negative changes in social and affective wellbeing. Applying a combination of frequentist and Bayesian multilevel models, we found that the effects of social media on wellbeing were heterogenous, but most people generally experienced a negative social media sensitivity in terms of changes to their loneliness, stress levels, affect and feelings of being accepted. People with psychologically vulnerable dispositions (e.g., those who were depressed, lonely, not satisfied with life) tended to experience a heightened negative social media sensitivity in comparison to people who were not psychologically vulnerable. People also experienced heightened negative social media sensitivities when in certain types of places (e.g., in social places, in nature) and while around certain types of people (e.g., around family members, close ties), as compared to using social media at home or when alone. Our results suggest that an understanding of the effects of social media on wellbeing should account for the psychological dispositions of social media users, and the physical and social contexts surrounding their use. We discuss theoretical and practical implications of social media sensitivity for scholars, policymakers, and those in the technology industry.