The aim of this study was to establish and evaluate a structural equation model to infer causal relationships among environmental and genetic factors on udder health. For this purpose, 537 Holstein Friesian cows were genotyped, and milk samples were analyzed for novel traits including differential somatic cell counts and specific mastitis pathogens. In the structural model, four latent variables (intramammary infection (IMI), production, time and genetics) were defined, which were explained using manifest measurable variables. The measurable variables included udder pathogens and somatic differential cell counts, milk composition, as well as significant SNP markers from previous genome-wide associations for major and minor pathogens. The housing system effect (i.e., compost-bedded pack barns versus cubicle barns) indicated a small influence on IMI with a path coefficient of −0.05. However, housing system significantly affected production (0.37), with ongoing causal effects on IMI (0.17). Thus, indirect associations between housing and udder health could be inferred via structural equation modeling. Furthermore, genotype by environment interactions on IMI can be represented, i.e., the detection of specific latent variables such as significant SNP markers only for specific housing systems. For the latent variable genetics, especially one SNP is of primary interest. This SNP is located in the EVA1A gene, which plays a fundamental role in the MAPK1 signaling pathway. Other identified genes (e.g., CTNNA3 and CHL1) support results from previous studies, and this gene also contributes to mechanisms of the MAPK1 signaling pathway.