Social perceptions of speakers are influenced by their voice information, including vocal characteristics and semantic content. Our study investigated how individuals’ warmth- and competence-related perceptions of speakers were affected by vocal pitch levels (i.e., high/low) and three kinds of semantic cues (i.e., prosocial, antisocial, and neutral) simultaneously. We have three key findings. First, antisocial cues negatively affected social perceptions, regardless of speakers’ gender. However, prosocial cues did not have positive impacts on evaluations of speakers because ratings were similar between prosocial cues and neutral cues. Second, female vocal pitch mattered for warmth-related perceptions but not for competence-related perceptions. The role of semantic cues should be additionally considered when investigating the impact of male vocal pitch on these perceptions. For example, higher-pitched men in prosocial contexts were perceived as warmer, while low-pitched men in antisocial contexts were judged as more competent. Third, the connection between vocal pitch and two kinds of perceptions showed an opposite trend, in which high pitch was related to more warmth but less competence, while the low pitch was associated with less warmth but more competence. These findings extend the understanding of the role of vocal pitch in the formation of stereotypes of strangers in different semantic contexts.