Implemented social distancing measures may have forestalled the spread of COVID-19, yet they suppressed the natural human need for contact. The aim of this systematic review was to explore the impact of the COVID-19 pandemic on adult sexual wellbeing and sexual behavior. An extensive search in Pubmed, Scopus, and PsycInfo databases based on PRISMA guidelines was conducted. After applying specific eligibility criteria, screening resulted in 38 studies. Results were drawn from 31,911 subjects and outlined the negative effect of the pandemic in sexual frequency, function, satisfaction, and the behavioral changes regarding masturbation and internet-based practices. Meta-analyses of the drawn data on 1,343 female, and 1,372 male subjects quantified the degree of sexual function change during the COVID-19 pandemic vs. prior the pandemic. A random effects model revealed the significant negative impact of the pandemic on female sexual function (SMD: 0.76, 95% CI:0.74 to 1.59), while no significant change was found for the males (SMD: 0.25, 95% Cl: −0.03 to 0.52). Significant heterogeneity was identified across included studies (p < 0.00001, I2 = 97%, I2 = 90% for females and males, respectively). As part of the global health, sexual wellbeing should be on the focus of clinicians and researchers.