Fragile X syndrome (FXS) is the most common form of inherited intellectual disability and the leading monogenetic cause of autism. One symptom of FXS and autism is sensory hypersensitivity (also called sensory over-responsivity). Perhaps related to this, the audiogenic seizure (AGS) is arguably the most robust behavioral phenotype in the FXS mouse model-the Fmr1 knockout (KO) mouse. Therefore, the AGS may be considered a mouse model of sensory hypersensitivity. Hyperactive circuits are hypothesized to underlie dysfunction in a number of brain regions in patients with FXS and Fmr1 KO mice, and the AGS may be a result of this. But the specific cell types and brain regions underlying AGSs in the Fmr1 KO are unknown. We used conditional deletion or expression of Fmr1 in different cell populations to determine whether Fmr1 deletion in those cells was sufficient or necessary, respectively, for the AGS phenotype in males. Our data indicate that Fmr1 deletion in glutamatergic neurons that express vesicular glutamate transporter 2 (VGlut2) and are located in subcortical brain regions is sufficient and necessary to cause AGSs. Furthermore, the deletion of Fmr1 in glutamatergic neurons of the inferior colliculus is necessary for AGSs. When we demonstrate necessity, we show that Fmr1 expression in either the larger population of VGlut2-expressing glutamatergic neurons or the smaller population of inferior collicular glutamatergic neurons-in an otherwise Fmr1 KO mouse-eliminates AGSs. Therefore, targeting these neuronal populations in FXS and autism may be part of a therapeutic strategy to alleviate sensory hypersensitivity.