The role of the oral microbiome in mental health has recently been appreciated within the proposed oral-brain axis. This study examined the structure and composition of the salivary microbiome in a large-scale population-based cohort of individuals reporting mental health symptoms (n = 306) compared to mentally healthy controls (n = 164) using 16S rRNA sequencing. Mental health symptoms were evaluated using validated questionnaires and included depression, anxiety, and posttraumatic stress disorder, with accompanying periodontal outcomes. Participants also indicated current or previous diagnoses of anxiety, depression, periodontitis, and gingivitis. Mental and periodontal health variables influenced the overall composition of the oral microbiome. PTSD symptoms correlated with reduced Haemophilus sputorum and elevated Prevotella histicola levels. Anxiety disorder diagnosis was associated with decreased Neisseria elongate and increased Oribacterium asaccharolyticum. P. histicola abundance was also positively associated with depressive scores and negatively associated with psychological quality of life. A higher abundance of Shuttleworthia and a lower abundance of Capnocytophaga were evident in those who reported a clinical periodontitis diagnosis. Functional prediction analysis revealed a potential role for tryptophan metabolism/degradation in the oral-brain axis, which was confirmed by lower plasma serotonin levels across symptomatic groups. Higher Eggerthia and lower Haemophilus parainfluenzae abundance were associated with reported clinical periodontitis diagnosis and psychotherapeutic efficacy. This study sheds light on the intricate interplay between oral microbiota, periodontal outcomes, and mental health, emphasizing the need for further exploration of the oral-brain axis to pave the way for novel therapeutic interventions and predicting therapeutic response.