Selectivity of cortical neurons for sensory stimuli can increase across days as animals learn their behavioral relevance, and across seconds when animals switch attention. While both phenomena are expressed in the same cortical circuit, it is unknown whether they rely on similar mechanisms. We imaged activity of the same neuronal populations in primary visual cortex as mice learned a visual discrimination task and subsequently performed an attention switching task. Selectivity changes due to learning and attention were uncorrelated in individual neurons. Selectivity increases after learning mainly arose from selective suppression of responses to one of the task relevant stimuli but from selective enhancement and suppression during attention. Learning and attention differentially affected interactions between excitatory and PV, SOM and VIP inhibitory cell classes. Circuit modelling revealed that cell class-specific top-down inputs best explained attentional modulation, while the reorganization of local functional connectivity accounted for learning related changes. Thus, distinct mechanisms underlie increased discriminability of relevant sensory stimuli across longer and shorter time scales.