Charged-particle spectra associated with direct photon (γ dir ) and π 0 are measured in p + p and Au + Au collisions at center-of-mass energy √ s NN = 200 GeV with the STAR detector at the Relativistic Heavy Ion Collider. A shower-shape analysis is used to partially discriminate between γ dir and π 0 . Assuming no associated charged particles in the γ dir direction (near side) and small contribution from fragmentation photons (γ frag ), the associated charged-particle yields opposite to γ dir (away side) are extracted. In central Au + Au collisions, the charged-particle yields at midrapidity (|η| < 1) and high transverse momentum (3 < p assoc T < 16 GeV/c) associated with γ dir and π 0 (|η| < 0.9, 8 < p trig T < 16 GeV/c) are suppressed by a factor of 3-5 compared with p + p collisions. The observed suppression of the associated charged particles is similar for γ dir and π 0 and independent of the γ dir energy within uncertainties. These measurements indicate that, in the kinematic range covered and within our current experimental uncertainties, the parton energy loss shows no sensitivity to the parton initial energy, path length, or color charge.