Various methods categorize tumour response after neoadjuvant therapy, including down-staging and tumour regression grading. Response categories allow comparison of different treatments within clinical trials and predict outcome. A reproducible response categorization could identify subgroups with high or low risk for the most appropriate subsequent treatments, like watch and wait. Lack of standardization and interpretation difficulties currently limit the usability of these approaches. In this review we describe these difficulties for the evaluation of chemoradiation in rectal cancer. An alternative approach of tumour response is based on patterns of residual disease, including fragmentation. We summarise the evidence behind this alternative method of response categorisation, which explains a number of very relevant clinical discrepancies. These issues include differences between downstaging and tumour regression, high local regrowth in advanced tumours during watchful waiting procedures, the importance of resection margins, the limited value of post-treatment biopsies and the relatively poor outcome of patients with a near complete pathological response. Recognition of these patterns of response can allow meaningful development of novel biomarkers in the future.