Several studies have suggested an overrepresentation of (symptoms of) autism spectrum disorder (ASD) among individuals with gender dysphoria. Three studies have taken the inverse approach in children with ASD and showed increased parent report of the wish to be of the opposite gender in this group. This study compared the self-reported wish to be of the opposite gender (one item of the Youth Self-Report [YSR] and the Adult Self-Report [ASR]) of 573 adolescents (469 assigned boys and 104 assigned girls) and 807 adults (616 assigned males and 191 assigned females) with ASD to 1016 adolescents and 846 adults from the general population. Emotional and behavioral problems were measured by the DSM-oriented scales of the YSR and ASR. In addition, the Children’s Social Behavior Questionnaire and the Adult Social Behavior Questionnaire were used to measure specific subdomains of the ASD spectrum to test whether specific subdomains of ASD were particularly involved. Significantly more adolescents (6.5%) and adults (11.4%) with ASD endorsed this item as compared to the general population (3–5%). In adolescents, assigned girls endorsed this item more than assigned boys. No significant gender differences were found in the adults with ASD. In addition, on all DSM-oriented scales of both the YSR and ASR, adolescents and adults with ASD who endorsed the gender item had significantly higher scores compared to those without. There were no significant associations between endorsement of the gender item and any specific subdomain of ASD, providing no evidence for a sole role of one of the ASD subdomains and endorsement of the wish to be the opposite gender.