Observers can simulate aspects of other people's tactile experiences. We asked whether they do so when faced with full-body social interactions, whether emerging representations go beyond basic sensorimotor mirroring, and whether they depend on processing goals and inclinations. In an EEG/ERP study, we presented line-drawn, dyadic interactions with and without affectionate touch. In an explicit and an implicit task, participants categorized images into touch vs no-touch and same vs opposite sex interactions, respectively. Modulations of central Rolandic rhythms implied that affectionate touch displays engaged sensorimotor mechanisms. Additionally, the late positive potential (LPP) being larger for images with as compared to without touch pointed to an involvement of higher order socio-affective mechanisms. Task and sex modulated touch perception. Sensorimotor responding, indexed by Rolandic rhythms, was fairly independent of the task but appeared less effortful in women than in men. Touch induced socio-affective responding, indexed by the LPP, declined from explicit to implicit processing in women and disappeared in men. In sum, this study provides first evidence that vicarious touch from full-body social interactions entails shared sensorimotor as well as socio-affective experiences. Yet, mental representations of touch at a socio-affective level are more likely when touch is goal relevant and observers are female. Together, these results outline the conditions under which touch in visual media may be usefully employed to socially engage observers.