We study finite groups that occur as combinatorial automorphism groups or geometric symmetry groups of convex polytopes. When Γ is a subgroup of the combinatorial automorphism group of a convex d-polytope, d ≥ 3, then there exists a convex d-polytope related to the original polytope with combinatorial automorphism group exactly Γ. When Γ is a subgroup of the geometric symmetry group of a convex d-polytope, d ≥ 3, then there exists a convex d-polytope related to the original polytope with both geometric symmetry group and combinatorial automorphism group exactly Γ. These symmetry-breaking results then are applied to show that for every abelian group Γ of even order and every involution σ of Γ, there is a centrally symmetric convex polytope with geometric symmetry group Γ such that σ corresponds to the central symmetry.