Neuroimaging studies have reported increased prefrontal cortex (PFC) activity during temporal context retrieval versus recognition memory. However, it remains unclear if these activations reflect PFC contributions to domain-general executive control processes or domain-specific retrieval processes. To gain a better understanding of the functional roles of these various PFC regions during temporal context retrieval we propose it is necessary to examine PFC activity across tasks from different domains, in which parallel manipulations are included targeting specific cognitive processes. In the current fMRI study, we examined domain-general and domain-specific PFC contributions to temporal context retrieval by increasing stimulus (but maintaining response number) and increasing response number (but maintaining stimulus number) across temporal context memory and ordering control tasks, for faces. The control task required subjects to order faces from youngest to oldest. Our behavioral results indicate that the combination of increased stimulus and response numbers significantly increased task difficulty for temporal context retrieval and ordering tasks. Across domains, increasing stimulus number, while maintaining response numbers, caused greater right lateral premotor cortex (BA 6/8) activity; whereas increasing response number, while maintaining stimulus number, caused greater domain-general left DLPFC (BA 9) and VLPFC (BA 44/45) activity. In addition, we found domain-specific right DLPFC (BA 9) activity only during retrieval events. These results highlight the functional heterogeneity of frontal cortex, and suggest its involvement in temporal context retrieval is related to its role in various cognitive control processes.