With the continued promise of immunotherapy as an avenue for treating cancer, understanding how host genetics contributes to the tumor immune microenvironment (TIME) is essential to tailoring cancer risk screening and treatment strategies. Using genotypes from over 8,000 European individuals in The Cancer Genome Atlas (TCGA) and 137 heritable tumor immune phenotype components (IP components), we identified and investigated 482 TIME associations and 475 unique TIME-associated variants. Many TIME-associated variants influence gene activities in specific immune cell subsets, such as macrophages and dendritic cells, and interact to promote more extreme TIME phenotypes. TIME-associated variants were predictive of immunotherapy response in human cohorts treated with immune-checkpoint blockade (ICB) in 3 cancer types, causally implicating specific immune-related genes that modulate myeloid cells of the TIME. Moreover, we validated the function of these genes in driving tumor response to ICB in preclinical studies. Through an integrative approach, we link host genetics to TIME characteristics, informing novel biomarkers for cancer risk and target identification in immunotherapy.