A central question in neuroscience is how context changes perception. In the olfactory system, for example, experiments show that task demands can drive divergence and convergence of cortical odor responses, likely underpinning olfactory discrimination and generalization. Here, we propose a simple statistical mechanism for this effect based on unstructured feedback from the central brain to the olfactory bulb, which represents the context associated with an odor, and sufficiently selective cortical gating of sensory inputs. Strikingly, the model predicts that both convergence and divergence of cortical odor patterns should increase when odors are initially more similar, an effect reported in recent experiments. The theory in turn predicts reversals of these trends following experimental manipulations and in neurological conditions that increase cortical excitability.