In contrast to the abundance of information on the many physiological and developmental actions of serotonin in molluscan nervous systems, comparatively little is known about the serotonin receptors involved in these responses. Embryos of the pulmonate gastropod, Helisoma trivolvis, display a cilia-driven rotational behavior that is regulated by endogenous serotonin. In the present study, two functional assays were used to determine some of the pharmacological properties of the receptors that mediate the cilio-excitatory action of serotonin. Time-lapse video microscopy was used to measure whole embryo rotation rate and cilia beat frequency in isolated cells. In dose-response experiments, serotonin was approximately 10 times more potent in stimulating cilia beat frequency over embryo rotation. In rotation experiments, 5-carboxyamidotryptamine and methysergide had effective agonist activity in dose ranges similar to that of serotonin (1 to 100 microM). In contrast, 8-hydroxydipropylaminotetralin HBr (8-OH-DPAT) displayed agonist activity of lower potency and effectiveness. Several compounds displayed antagonist activity in the 1 to 100 microM dose range, including mianserin, spiperone, ritanserin, 1-(1-naphthyl)piperazine, and propranolol. alpha-Methylserotonin had mixed agonist-antagonist activity, and metoclopramide, MDL-72222, and ketanserin were inactive. Experiments on isolated cells suggested that the extremely effective antagonism displayed by mianserin in the embryo rotation assay was due to its specific activity at ciliary serotonin receptors. These results implicate the presence of a novel serotonin receptor on embryonic ciliated cells that is pharmacologically distinct from those previously characterized in vertebrate or invertebrate systems.