Photon correlation measurements reveal memory effects in the optical emission of single InAs quantum dots with timescales from 10 to 800 ns. With above-band optical excitation, a longtimescale negative correlation (antibunching) is observed, while with quasi-resonant excitation, a positive correlation (blinking) is observed. A simple model based on long-lived charged states is presented that approximately explains the observed behavior, providing insight into the excitation process. Such memory effects can limit the internal efficiency of light emitters based on single quantum dots, and could also be problematic for proposed quantum-computation schemes.