Gastric cancer (GC) is one of the most common types of malignancy worldwide and is accompanied by both high mortality and morbidity rates. Homeobox B13 (HOXB13) has been reported to act as a tumor suppressor gene in multiple types of human cancer. The present study aimed to investigate the effects and potential underlying molecular mechanisms of HOXB13 in the progression of GC. The expression of HOXB13 in GC cells was first examined using the Cancer Cell Line Encyclopedia database and subsequently validated in a number of GC cell lines. Following HOXB13 overexpression (Ov-HOXB13), HGC-27 cell proliferation was evaluated by colony formation and Cell Counting Kit-8 assays. Wound healing and Matrigel assays were used to determine the migratory and invasive abilities, respectively. Additionally, cell apoptosis was assessed using TUNEL staining, and the expression of apoptosis-related proteins was detected by western blot analysis. Subsequently, TEA domain transcription factor 4 (TEAD4) was overexpressed to evaluate the effects on HGC-27 cell proliferation, migration, invasion and apoptosis following co-transfection with Ov-HOXB13. The potential binding sites of HOXB13 on the vestigial-like family member 4 (VGLL4) promoter were verified using chromatin immunoprecipitation and dual luciferase reporter assays. Moreover, the expression levels of proteins involved in the Hippo signaling pathway were analyzed using western blotting. The results revealed that the expression of HOXB13 was notably lower in GC cells compared with normal gastric cells. The overexpression of HOXB13 significantly inhibited the proliferation, migration and invasion, but promoted the apoptosis of HGC-27 cells. Moreover, Ov-HOXB13 downregulated TEAD4 expression. Notably, Ov-TEAD4 transfection partially reversed the effects of Ov-HOXB13 on the cellular behaviors of HGC-27 cells. HOXB13 was also confirmed to bind with the VGLL4 promoter. The knockdown of VGLL4 restored the inhibitory effects of Ov-HOXB13 on the expression levels of VGLL4 and Hippo pathway signaling proteins. In conclusion, the findings of the present study suggested that Ov-HOXB13 may suppress the proliferation, migration and invasion, and promote the apoptosis of GC cells through the transcriptional activation of VGLL4 to inhibit the involvement of TEAD4 in the Hippo signaling pathway. These results may provide novel and potent targets for the treatment of GC.