Many autistics report being distressed by eye contact, but eye-tracking studies suggest that eye contact is associated with hypo-arousal rather than hyper-arousal in autism. Within a live face-to-face paradigm combining a wearable eye-tracker with electrodermal activity sensors, 80 adults (40 autistics) defined words in front of an experimenter either staring at their eyes (direct gaze condition) or looking elsewhere (averted gaze condition). Autistics did not differ from neurotypicals in their eye behaviours nor their skin conductance responses. Autistics did not appear distressed when they were looking at the experimenter’s eyes in the direct gaze condition. However, neurotypicals, compared to autistics, might experience more stress when looking at the experimenter in the averted gaze condition, even after controlling for social anxiety and alexithymia. In comparison to autistics, neurotypicals might be hyper-aroused when they look at someone avoiding eye contact. Based on a bidirectional perspective on interactional difficulties in autism, we speculate that the neurotypicals’ distress when their attempts to eye contact are not reciprocated could make their behaviour insistent, which, in turn, could make the autistics uncomfortable. In our study, participants’ partner remained passive, displaying no specific reaction when a mutual gaze was shared or not. Future studies should test different partner reactions to gaze in various social contexts. Lay abstract What is already known about the topic? Autistics are usually reported to share less eye contact than neurotypicals with their interlocutors. However, the reason why autistics might pay less attention to eyes looking at them is still unknown: some autistics express being hyper-aroused by this eye contact, while some eye-tracking studies suggest that eye contact is associated with hypo-arousal in autism. What this paper adds? This study is based on a highly controlled live face-to-face paradigm, combining a wearable eye-tracker (to study eye behaviours) with electrodermal activity sensors (to assess potential stress). We draw a nuanced picture of social attention in autism, as our autistic participants did not differ from our neurotypical group in their eye behaviours nor their skin conductance responses. However, we found that neurotypicals, compared to autistics, seemed to be much more distressed when their interlocutor did not gaze at them during the experiment. Implications for practice, research or policy: Our study encourages to consider social interaction difficulties in autism as a relational issue, instead as an individual deficit. This step might be first taken in research, by implementing paradigms sensitive to the experimenter’s role and attitude.