Tumor budding has been found to be of prognostic significance for several cancers, including colorectal cancer (CRC). Additionally, the molecular classification of CRC has led to the identification of different immune microenvironments linked to distinct prognosis and therapeutic response. However, the association between tumor budding and the different molecular subtypes of CRC and distinct immune profiles have not been fully elucidated. This study focused, firstly, on the validation of derived xenograft models (PDXs) for the evaluation of tumor budding and their human counterparts and, secondly, on the association between tumor budding and the immune tumor microenvironment by the analysis of gene expression signatures of immune checkpoints, Toll-like receptors (TLRs), and chemokine families. Clinical CRC samples with different grades of tumor budding and their corresponding PDXs were included in this study. Tumor budding grade was reliably reproduced in early passages of PDXs, and high-grade tumor budding was intimately related with a poor-prognosis CMS4 mesenchymal subtype. In addition, an upregulation of negative regulatory immune checkpoints (PDL1, TIM-3, NOX2, and IDO1), TLRs (TLR1, TLR3, TLR4, and TLR6), and chemokine receptors and ligands (CXCR2, CXCR4, CXCL1, CXCL2, CXCL6, and CXCL9) was detected in high-grade tumor budding in both human samples and their corresponding xenografts. Our data support a close link between high-grade tumor budding in CRC and a distinctive immune-suppressive microenvironment promoting tumor invasion, which may have a determinant role in the poor prognosis of the CMS4 mesenchymal subtype. In addition, our study demonstrates that PDX models may constitute a robust preclinical platform for the development of novel therapies directed against tumor budding in CRC.