Many previous studies have shown that background color affects the discriminability and appearance of color stimuli. However, research on visual search has not typically considered the role that the background may play. Rosenholtz (2001a) has suggested that color search asymmetries result from the relationship between the stimuli and the background. Here we test the hypothesis that background color should have an effect on asymmetries in visual search based on color, using searches for color stimuli on different colored backgrounds. Observers searched for a single known target stimulus among homogeneous distractor stimuli. The target stimulus differed from the distractors only in chromaticity, but targets and distractors both differed from the backgrounds in luminance so that they were easily visible regardless of chromaticity. Target/distractor pairs differed primarily in saturation (Experiments 1, 2, & 3) or in hue (Experiment 4). Each member of each pair of colors served as target and distractor color on both achromatic and red backgrounds. When the stimuli were presented on an achromatic background, response times were shorter when the more saturated member of each pair of colors served as the target color. When the same stimuli were presented on a red background, the asymmetry was either reversed or abolished. When target and distractors differed in hue, there was little asymmetry on the achromatic background but a sizable asymmetry for some color pairs on the red background. On both backgrounds, the magnitude of the asymmetry varied with the difference between the stimulus colors and the background color. Results confirm that asymmetries in color search are dependent on the relationship between the stimulus colors and the background color. Two candidate models are suggested that show promise in predicting these experimental results: Rosenholtz' saliency model (1999, 2001a) and a modification to signal detection theory models in which the observation noise is proportional to the difference between target/distractor color and background color.