Of the five cloned muscarinic receptor subtypes, dopamine (DA) neurons in the substantia nigra and ventral tegmental areas have been shown to be selectively enriched with the mRNA for the m5 subtype, suggesting that muscarinic modulation of DA neurons may have a distinct pharmacology. In the present study we have used dissociated cell cultures of fetal rat ventral mesencephalon to characterize muscarinic modulation of DA neurons. [3H]DA release stimulated by activation of N-methyl-D-aspartate (NMDA) receptors was potentiated by carbachol, a mixed muscarinic-nicotinic agonist, and by oxotremorine-M, a muscarinic agonist. Neither carbachol nor oxotremorine-M had an effect on [3H]DA release evoked by the non-NMDA agonists, kainate or quisqualate. A nicotinic agonist, DMPP, had no effect on NMDA-stimulated release. Potentiation of NMDA-stimulated [3H]DA release by oxotremorine-M was inhibited by the broad spectrum muscarinic antagonist, QNB, and by low concentrations of a putative M1 antagonist, pirenzepine, while much higher concentrations of a purported M2 antagonist, AF-DX 384, were required to reverse the oxotremorine-M effect. The muscarinic antagonist, 4-DAMP, was active in a concentration range between that required for pirenzepine and AF-DX 384. Further experiments examined intracellular messenger mechanisms coupled to the muscarinic receptors modulating NMDA-stimulated [3H]DA release. In contrast to oxotremorine-M, two muscarinic agents with only weak partial agonism with respect to phosphoinositide turnover, pilocarpine and arecoline, had no effect on NMDA-stimulated [3H]DA release.(ABSTRACT TRUNCATED AT 250 WORDS)