An elevated serum bilirubin has been reported to be associated with a reduced risk of some cancer; however, the prognostic significance of serum bilirubin in colorectal cancer wasn’t fully understood. The purpose of this study was to evaluate whether serum bilirubin could predict the prognosis of patients in stages II and III colorectal cancer. A retrospective cohort of 986 patients with colorectal cancer who received surgical resection between January 2005 and December 2010 was included in the study. Levels for serum bilirubin were obtained from medical records. Survival analysis was used to evaluate the predictive value of bilirubin. Serum direct bilirubin (DBIL) was validated as a significant prognostic factor by univariate cox regression test for both overall survival (OS) and disease free survival (DFS) (P < 0.05). X-tile program identified 3.6 as optimal cutoff values for DBIL in terms of OS and DFS. Patients were then divided into DBIL high (DBIL ≥ 3.60 μmol/l) and low group (DBIL < 3.60 μmol/l) according to the optimal cutoff. High DBIL had higher percentage of lymph node metastasis and lymphovascular invasion as compared with low DBIL levels (P < 0.05). Multivariate cox regression analyses confirmed that high DBIL level was an independently prognostic factor for both OS (HR: 1.337, 95% CI: 1.022–1.748, P = 0.034) and DFS (HR: 1.312, 95% CI: 1.049–1.643, P = 0.018). In addition, nomograms on OS and DFS were established according to all significant factors, and c-indexes were 0.715 (95% CI: 0.683–0.748) and 0.704 (95% CI: 0.678–0.730), respectively. Nomograms based on OS and DFS can be recommended as practical models to evaluate prognosis for CRC patients.