Introductions of invasive species to new environments often result in rapid rates of trait evolution. While in some cases these evolutionary transitions are adaptive and driven by natural selection, they can also result from non-adaptive processes associated with the invasion history. Here, we examined the role of adaptive and non-adaptive evolutionary processes in the brown anole (Anolis sagrei), a widespread invasive lizard for which genetic data have helped trace the sources of non-native populations. We focused on the dewlap, a signaling trait known to be subject to multiple selective pressures. We measured dewlap reflectance, pattern, and size in non-native populations across the southeastern United States. We combine these trait measurements with quantification of environmental variables known to influence dewlap signal effectiveness, such as canopy openness. Further, we use genome-wide data to estimate ancestry and to perform association mapping for dewlap traits. We found that among-population variation in dewlap characteristics is best explained by ancestry, as contributed by invasion history. This result was supported by genome-wide association mapping, which identified several ancestry-specific loci associated with dewlap traits. Despite the strong imprint of invasion history on dewlap variation, we also detect significant relationships between dewlap traits and local environmental conditions. Thus, our results are also consistent with natural selection influencing trait evolution during the brown anole invasion. Our study clarifies the importance of ancestry and admixture in shaping phenotypes during biological invasion, while also showing that some traits can respond adaptively to conditions encountered in the invasive range despite potential constraints imposed by invasion history.