Fourier transform infrared (FTIR) spectroscopy is widely used to determine milk composition. In this study, 1,060 milk infrared wavenumbers ranging from 925 to 5,008cm(-1) of 1,748 Holstein Friesian cows on 371 herds in the Netherlands were available. The extent to which infrared wavenumbers are affected by genetic and environmental factors was investigated. Inter-herd heritabilities of 1,060 infrared wavenumbers ranged from 0 to 0.63, indicating that the genetic background of infrared wavenumbers differs considerably. The majority of the wavenumbers have moderate to high inter-herd heritabilities ranging from 0.20 to 0.60. The diacylglycerol O-acyltransferase 1 (DGAT1), stearoyl-CoA desaturase (SCD1), κ-casein (CSN3), and β-lactoglobulin (LGB) polymorphisms are known to have a large effect on milk composition, and therefore we studied the effects of these polymorphisms on infrared wavenumbers. The DGAT1 polymorphism had highly significant effects on many wavenumbers. In contrast, the SCD1 polymorphism did not significantly affect any of the wavenumbers. The SCD1 is known to have a strong effect on the content of C10:1, C12:1, C14:1, and C16:1 fatty acids. Therefore, these results suggest that FTIR spectra contain little direct information on these monounsaturated fatty acids. The CSN3 and LGB polymorphisms had significant effects on a few wavenumbers. Differences between herds explained 10 to 25% of the total variance for most wavenumbers. This suggests that the wavenumbers of milk FTIR spectra are indicative for differences in feeding and management between herds. The wavenumbers between 1,619 and 1,674cm(-1) and between 3,073 and 3,667cm(-1) are strongly influenced by water absorption and usually excluded when setting up prediction equations. However, we found that some of the wavenumbers in the water absorption region are affected by the DGAT1 polymorphism and lactation stage. This suggests that these wavenumbers contain useful information regarding milk composition.