Vitamin D insufficiency has become a common health problem worldwide, particularly among pregnant women and young children. Therefore, we sought to identify environmental, dietary, and genetic determinants of serum 25(OH)-vitamin D (25(OH)D) levels during pregnancy and early childhood. 25(OH)D was measured in women at 24-weeks of gestation (n = 738) and one-week postpartum (n = 284) in the population-based Danish COPSAC2010 mother–child cohort; and in cord blood (n = 257) and age 4 years (n = 298) in children from the at-risk COPSAC2000 mother–child cohort. Environmental, dietary, and genetic variables were tested for association with 25(OH)D using linear regression analyses. After adjusting for season of blood sampling, determinants of lower 25(OH)D levels during pregnancy in the women were higher pre-pregnancy BMI, lower age at birth, lower genetic vitamin D score, lower dietary vitamin D intake, and lower social circumstances. In children, the determinants were lower maternal age at birth, higher pre-pregnancy BMI, lower genetic vitamin D score, older siblings, exposure to tobacco smoking, and female sex. Genetics was an important determinant at all time points, alone explaining 2%–11% of the variance in 25(OH)D. Important determinants of circulating 25(OH)D levels during pregnancy and early childhood include environmental factors, diet, and to a large extent genetics.