We investigated whether lateralized BOLD-fMRI activations in Broca's region, Wernicke's region and visual word form area (VWFA) reflect task shift costs and to which extent these effects are specific to language related task shifts. We employed a linguistic one-back memory paradigm where participants (n = 58) on each trial responded to whether a given word was the same as the previous word. In concordance with previous findings we found that conceptual shifts (CS), i.e. new words, elicited a strongly left-lateralized response in all three regions compared to repeat words. Words were sometimes presented through the visual modality (read) and sometimes through the auditory modality (spoken). This enabled the study of perceptual modality shifts (PS) relative to trials that stayed in the same modality as the previous trials. Again, we found a strongly left-lateralized effect in all regions. This was independent of whether the word was a CS or not, suggesting that linguistic translation across modalities taxes the same system as CS. Response shifts (RS), on the other hand, when shifting from one response (e.g. reporting a repeat word) to another (e.g. reporting a new word) did not yield an observable left lateralized response in any of the regions, suggesting that the lateralized task shift cost effects in these regions are not shared by all types of task shifts. Lateralization for individual tasks was found to be correlated across brain regions, but not across tasks, suggesting that lateralization may not be a unitary phenomenon, but vary across participants according to task demands. Both response time and lateralization were found to reflect the demands not only of the current trial but also of the previous trial, illustrating the context dependency of even simple cognitive tasks.