In the cerebral cortex, the selectivity of neurons for features of sensory stimuli arises through the interaction of excitatory and inhibitory synaptic inputs. Excitatory neurons receive inhibitory input that closely tracks excitation 1-4 , stabilizing network dynamics 5 while improving efficiency and robustness of the neural code 6-8 . However, how this balance of excitation and inhibition is achieved by cortical circuits is unclear, since inhibitory interneurons are thought to pool the inputs of nearby excitatory cells and provide them with non-specific inhibition proportional to the activity of the local network 9-13 . Here we show that although parvalbumin-expressing (PV) inhibitory cells make connections with the majority of nearby pyramidal cells, the strength of their synaptic connections is structured according to the similarity of the cells' responses. Individual PV cells strongly inhibit those pyramidal cells that provide them with strong excitation and share their visual selectivity. This fine-tuning of synaptic weights supports co-tuning of inhibitory and excitatory inputs onto individual pyramidal cells despite dense connectivity between inhibitory and excitatory neurons. Our results indicate that individual PV cells are preferentially integrated into subnetworks of inter-connected, co-tuned pyramidal cells, stabilising their recurrent dynamics. Conversely, weak but dense inhibitory connectivity between subnetworks is sufficient to support competition between them, de-correlating their output. We suggest that the history and structure of correlated firing adjusts the weights of both inhibitory and excitatory connections, supporting stable amplification and selective recruitment of cortical subnetworks.