The genetic architecture of variation in evolutionary fitness determines the trajectory of adaptive change. We identified quantitative trait loci (QTL) affecting fitness in a mapping population of recombinant inbred lines (RILs) derived from a cross between moist-and dry-associated ecotypes of Avena barbata. We estimated fitness in 179 RILs in each of two natural environments in each of 4 years. Two loci account for over half of the variation in geometric mean fitness across environments. These loci are associated in repulsion phase in the wild ecotypes, suggesting the potential for strong transgressive segregation, but also show significant epistasis giving hybrid breakdown. This epistasis is the result of sharply lower fitness in only one of the recombinant genotypes, suggesting that the loci may contain synergistically acting mutations. Within each trial (year/site combination), we can explain less of the variation than for geometric mean fitness, but the two major loci are associated with variation in fitness in most environments. Tests for pleiotropic effects of QTL on fitness in different environments reveal that the same loci are under selection in all trials. Genotype-by-environment interactions are significant for some loci, but this reflects variation in the strength, not the direction of selection.