Background Both tumor deposits (TD) and perineural invasion (PNI) have been identified as risk factors for poor survival in patients with non-metastatic colorectal adenocarcinoma (CRC). However, the adverse impacts of TD and PNI on the survival of patients with non-metastatic CRC have not been compared. Method Patients with non-metastatic CRC with known TD and PNI status were selected from the Surveillance, Epidemiology, and End Results (SEER) database. First, bivariate logistic regression analysis was utilized to identify the factors associated with TD and PNI status. Then, patients were divided into four groups, according to TD and PNI status. Propensity score matching (PSM) was performed to balance the baseline covariates. The impact of TD and PNI on survival was assessed by analyzing overall survival (OS) and cancer-specific mortality (CSM) rates. OS was calculated by the Kaplan–Meier method with log-rank analysis. CSM was estimated by competing risk analysis using the Fine and Gray model. Results A total of 70 689 patients with CRC met the inclusion and exclusion criteria. The positive rates of TD and PNI were 9.37% and 9.91%, respectively. For TD, the most important risk factor was N stage. With respect to PNI, the most significant factor was T stage. Tumor location, tumor size, differentiation grade, and serum CEA level were also correlated with TD and PNI status. After PSM, 1849 pairs were selected. Patients with TD+PNI+ status had the worst 5 year CSM and 5 year OS. In addition, the long-term survival outcomes of patients with TD+PNI− and TD−PNI+ status were comparable. Conclusion The adverse impacts of TD and PNI on the survival of patients with non-metastatic CRC were comparable. CRC patients with both TD and PNI positive had the worst survival outcome.