The success of cancer immunotherapy relies on the knowledge of the tumor microenvironment and the immune evasion mechanisms in which the tumor, stroma, and infiltrating immune cells function in a complex network. The potential barriers that profoundly challenge the overall clinical outcome of promising therapies need to be fully identified and counteracted. Although cancer immunotherapy has increasingly been applied, we are far from understanding how to utilize different strategies in the best way and how to combine therapeutic options to optimize clinical benefit. This review intends to give a contemporary and detailed overview of the different roles of immune cells, exosomes, and molecules acting in the tumor microenvironment and how they relate to immune activation and escape. Further, current and novel immunotherapeutic options will be discussed.