Enlargement of the aorta is an important risk factor for aortic aneurysm and dissection, a leading cause of morbidity in the developed world. While Mendelian genetics account for a portion of thoracic aortic disease, the contribution of common variation is not known. Using standard techniques in computer vision, we performed automated extraction of Ascending Aortic Diameter (AsAoD) from cardiac MRI of 36,021 individuals from the UK Biobank. A multi-ethnic genome wide association study and trans-ethnic meta-analysis identified 99 lead variants across 71 loci including genes related to cardiovascular development (HAND2, TBX20) and Mendelian forms of thoracic aortic disease (ELN, FBN1). A polygenic risk score predicted prevalent risk of thoracic aortic aneurysm within the UK Biobank (OR 1.50 per standard deviation (SD) polygenic risk score (PRS), p=6.30x10-03) which was validated across three additional biobanks including FinnGen, the Penn Medicine Biobank, and the Million Veterans Program (MVP) in individuals of European descent (OR 1.37 [1.31 - 1.43] per SD PRS), individuals of Hispanic descent (OR 1.40 [1.16 - 1.69] per SD PRS, p=5.6x10-04), and individuals of African American descent (OR 1.08 [1.00 - 1.18] per SD PRS, p=0.05). Within individuals of European descent who carried a diagnosis of thoracic aneurysm, the PRS was specifically predictive of the need for surgical intervention (OR 1.57 [1.15 - 2.15] per SD PRS, p=4.45x10-03). Using Mendelian Randomization our data highlight the primary causal role of blood pressure in reducing dilation of the thoracic aorta. Overall our findings link normal anatomic variation to extremes observed in Mendelian syndromes and provide a roadmap for the use of genetic determinants of human anatomy in both understanding cardiovascular development while simultaneously improving prediction and prevention of human disease.