Rare damaging variants in a large number of genes are known to cause monogenic developmental disorders (DD), and have been shown to cause milder sub-clinical phenotypes in population cohorts. To investigate potential genetic modifiers, we identified individuals in UK Biobank with predicted deleterious variants in 599 autosomal dominant DD genes, and found that carrying multiple rare variants in these genes had an additive adverse effect on numerous cognitive and socio-economic traits, which could be partially counterbalanced by a higher educational attainment polygenic score (EA-PGS). Amongst rare DD variant carriers, those with a DD-related clinical diagnosis had a substantially lower EA-PGS and more severe phenotype than those without. Our results suggest that the overall burden of both rare and common variants can modify the expressivity of a phenotype, which may influence whether an individual reaches the threshold for clinical disease.