Several authors have suggested a deep symmetry between the psychological processes that underlie our ability to remember the past and make predictions about the future. The judgment of recency (JOR) task measures temporal order judgments for the past by presenting pairs of probe stimuli; participants choose the probe that was presented more recently. We performed a short-term relative JOR task and introduced a novel judgment of imminence (JOI) task to study temporal order judgments for the future. In the JOI task, participants were trained on a probabilistic sequence. During a test phase, the sequence was occasionally interrupted with pairs of probes. Participants chose the probe that they expected would be presented sooner. Replicating prior work, we found that in JOR the correct RT depended only on the recency of the more recent probe. This suggests that memory for the past was supported by a backward self-terminating search model operating on a temporally-organized representation of the past. Analogously, in the JOI task we find that correct RT depended only on the imminence of the more imminent probe. By analogy to the JOR results, this suggests a forward self-terminating search model operating on a temporally-organized representation of the future. Critically, in both JOR and JOI the increase in RT with recency/imminence was sublinear, suggesting that both the timeline for the future and the timeline of the past are compressed. These results place strong constraints on computational models for constructing the predicted future.