Over the course of a lifetime, adults develop perceptual categories for the vowels and consonants in their native language, based on the distribution of those sounds in their environment. However, in any given listening situation, the short-term distribution of sounds can cause changes in this long-term categorization. For example, if the same sound (the "adaptor") is heard many times in a short period of time, listeners adapt and become less prone to hearing that sound. Although hundreds of speech selective adaptation experiments have been published, there is almost no information about how long this adaptation lasts.Using stimuli chosen to produce very large initial adaptation, we test adaptation effects with essentially no delay, and with delays of 25 min, 90 min, and 5.5 hr; these tests probe the duration of adaptation both in the (single) ear to which the adaptor was presented, and in the opposite ear. Reliable adaptation remains 5.5 hours after exposure in the same-ear condition, whereas it is undetectable at 90 min in the opposite ear. Surprisingly, the amount of residual adaptation is largely unaffected by whether the listener is exposed to speech between adaptation and test, unless the speech shares critical acoustic properties with the adapting sounds. Analyses of the shifts on three time scales (seconds, minutes, and hours) provide information about the multiple levels of analysis that the speech signal undergoes. Public Significance Statement: Spoken language is the primary mode of communication for most humans. The current work helps to specify the ways in which the speech signal is decoded, helping us to understand how humans accomplish this challenging task.