Background Despite being a promising strategy, current chemotherapy for gastric cancer (GC) is limited due to adverse side effects and poor survival rates. Therefore, new drug-delivery platforms with good biocompatibility are needed. Recent studies have shown that nanoparticle-based drug delivery can be safe, eco-friendly, and nontoxic making them attractive candidates. Here, we develop a novel selenium-nanoparticle based drug-delivery agent for cancer treatment from plant extracts and selenium salts. Results Selenium cations were reduced to selenium nanoparticles using Kaempferia parviflora (black ginger) root extract and named KP-SeNP. Transmission electron microscopy, selected area electron diffraction, X-ray diffraction, energy dispersive X-ray, dynamic light scattering, and Fourier-transform infrared spectrum were utilized to confirm the physicochemical features of the nanoparticles. The KP-SeNPs showed significant cytotoxicity in human gastric adenocarcinoma cell (AGS cells) but not in normal cells. We determined that the intracellular signaling pathway mechanisms associated with the anticancer effects of KP-SeNPs involve the upregulation of intrinsic apoptotic signaling markers, such as B-cell lymphoma 2, Bcl-associated X protein, and caspase 3 in AGS cells. KP-SeNPs also caused autophagy of AGS by increasing the autophagic flux-marker protein, LC3B-II, whilst inhibiting autophagic cargo protein, p62. Additionally, phosphorylation of PI3K/Akt/mTOR pathway markers and downstream targets was decreased in KP-SeNP-treated AGS cells. AGS-cell xenograft model results further validated our in vitro findings, showing that KP-SeNPs are biologically safe and exert anticancer effects via autophagy and apoptosis. Conclusions These results show that KP-SeNPs treatment of AGS cells induces apoptosis and autophagic cell death through the PI3K/Akt/mTOR pathway, suppressing GC progression. Thus, our research strongly suggests that KP-SeNPs could act as a novel potential therapeutic agent for GC. Graphical Abstract