We examine a common origin of four-dimensional flavor, CP, and U (1) R symmetries in the context of heterotic string theory with standard embedding. We find that flavor and U (1) R symmetries are unified into the Sp(2h + 2, C) modular symmetries of Calabi-Yau threefolds with h being the number of moduli fields. Together with the Z CP 2 CP symmetry, they are enhanced to GSp(2h+2, C) ≃ Sp(2h+2, C)⋊Z CP 2 generalized symplectic modular symmetry. We exemplify the S 3 , S 4 , T ′ , S 9 non-Abelian flavor symmetries on explicit toroidal orbifolds with and without resolutions and Z 2 , S 4 flavor symmetries on three-parameter examples of Calabi-Yau threefolds. Thus, non-trivial flavor symmetries appear in not only the exact orbifold limit but also a certain class of Calabi-Yau threefolds. These flavor symmetries are further enlarged to non-Abelian discrete groups by the CP symmetry.