Horizontal Gene Transfer (HGT) is a powerful evolutionary force facilitating bacterial adaptation and emergence of novel phenotypes. Several factors, including environmental ones, are predicted to restrict HGT, but we lack data supporting these predictions. Here, we address this gap by measuring the relative fitness of 44 genes horizontally transferred from E. coli to S. Typhimurium strain 4/74 in four infection relevant environments and estimated the distribution of fitness effects (DFE) in each environment. Roughly half of the genes, 40-50%, had significant fitness effects in at least one environment, with most genes showing significant gene by environment (G x E) interactions. We further identified 12 genes with dosage-dependent effects across the four environments, indicating that intolerance of the recipient strain to high protein dosage is a significant barrier to HGT. We also found longer genes had stronger fitness detriments than shorter ones, showing that gene length can significantly hinder HGT. In summary, a substantial fraction of transferred genes had a significant fitness cost on the recipient, with both gene characteristics and the environment providing barriers to HGT.