From a conditional adaptation vantage point, early life caregiving adversity likely enhances aspects of cognition needed to manage interpersonal threats. Yet, research examining early life care and offspring cognition predominantly relies upon experiments including affectively neutral stimuli, with findings generally interpreted as “early-life caregiving adversity is, de facto, ‘bad’ for cognitive performance.” Here, in a Southeast Asian sample, we examined observed maternal sensitivity in infancy and cognitive performance 3 years later as preschoolers took part in three tasks, each involving both a socioemotional (SE) and non-socioemotional (NSE) version: relational memory (n = 236), cognitive flexibility (n = 203), and inhibitory control (n = 255). Results indicate the relation between early life caregiving adversity and memory performance significantly differs (Wald test = 7.67, (1), P = 0.006) depending on the SE versus NSE context, with maternal sensitivity in infancy highly predictive of worse memory for SE stimuli, and amongst girls, also predictive of better memory when NSE stimuli are used. Results concerning inhibitory control, as well as cognitive flexibility in girls, also tentatively suggest the importance of considering the SE nature of stimuli when assessing relations between the caregiving environment and cognitive performance. As not all approaches to missing data yielded similar results, implications for statistical approaches are elaborated. We conclude by considering how an adaptation-to-context framework approach may aid in designing pedagogical strategies and well-being interventions that harness pre-existing cognitive strengths.