Helmet use on ski slopes has steadily increased worldwide over the past years. A common reason reported for helmet non-use, however, is impaired hearing. Therefore, an intra-subject design study was conducted to compare hearing thresholds and sound source localization of 21 adults with normal hearing in an anechoic chamber when wearing a ski helmet and ski goggles or wearing a ski cap and ski goggles to the condition head bare. Hearing thresholds while wearing a ski helmet (6.8 ± 1.6 dB HL) and ski cap (5.5 ± 1.6 dB HL) were significantly different (p = 0.030, d = 0.44). Compared to head bare (2.5 ± 1.2 dB HL), a significant difference was found for the ski helmet only (p = 0.040, d = 1.57). Regarding sound source localization, correct scores in the condition head bare (90%) showed a highly significant difference compared with those of condition cap (65%) and helmet (58%), respectively (p < 0.001; d > 2.5). Compared to the ski cap, wearing the helmet significantly reduced correct scores (p = 0.020, d = 0.59) irrespective of the tested sound pressure levels. In conclusion, wearing a ski helmet impairs hearing to a small though significantly greater extent compared with a cap, the degree, however, being less than what is termed as a hearing impairment. Compared to the condition head bare, wearing a ski cap or a ski helmet significantly reduced one's ability of sound source localization.