Xenopus laevis oocytes were used to analyse the effects of serotonin (5-hydroxytryptamine, 5-HT) and serotonergic agents on ionic currents elicited by the activation of mammalian muscle nicotinic acetylcholine receptors (AChRs). 5-HT as well as other serotonergic agents, such as ketanserin, 8-hydroxy-2-(di-n-propylamino)tetralin (8-OH-DPAT), methysergide, spiperone, or fluoxetine alone (up to 1 mM), did not elicit membrane currents in Xenopus oocytes expressing AChRs, but they reversibly reduced the current elicited by acetylcholine (ACh-current). Serotonin was applied before, together with or after ACh application, and its effects were examined on desensitizing and non-desensitizing ACh-currents. 5-HT reduced the amplitude and accelerated the desensitization of the desensitizing currents. In contrast, non-desensitizing currents were reduced in amplitude but their time course was not significantly affected. With the same concentration of 5-HT the inhibition was stronger on desensitizing than on non-desensitizing ACh-currents. For example, 100 microM 5-HT reduced the peak of a desensitizing ACh-current to 0. 48 +/- 0.06 (peak current ratio) and after 40 s the current was reduced to a ratio of 0.25 +/- 0.04, whereas a non-desensitizing ACh-current was reduced to a ratio of 0.66 +/- 0.01. All the serotonergic agents tested inhibited the ACh-currents rapidly and reversibly, suggesting that they are acting directly on the AChRs. The half-inhibitory concentration, IC50, of 5-HT acting on non-desensitizing currents elicited by 250 nM ACh was 247 +/- 26 microM and the Hill coefficient was 0.88, suggesting a single site for the interaction of 5-HT with the receptor. It appears that 5-HT inhibits AChRs non-competitively because neither the half-effective concentration of ACh, EC50, for ACh-current nor the Hill coefficient were affected by 5-HT. Furthermore, the extent of inhibition of 5-HT on AChRs did not depend on the nicotinic agonist (suberyldicholine, ACh or nicotine). The inhibition of AChRs by serotonergic agents was voltage-dependent. The electrical distance of the binding site for 5-HT was 0.75, whereas for the other serotonergic agents tested it was 0.22, suggesting that ketanserin, 8-OH-DPAT, methysergide, spiperone and fluoxetine act within the ion channel, but at a site more external than that for 5-HT. These substances inhibited the ACh-current more potently than 5-HT. We conclude that 5-HT and serotonergic agents inhibit, in a non-competitive manner, the ACh-current in muscle AChRs by blocking the open receptor-channel complex. Moreover, 5-HT appears to promote the desensitized state of the receptor when the current is elicited by high ACh concentrations.