T cells are important players in the antitumor immune response. Over the past few years, the adoptive transfer of genetically modified, autologous T cells—specifically redirected toward the tumor by expressing either a T cell receptor (TCR) or a chimeric antigen receptor (CAR)—has been adopted for use in the clinic. At the moment, the therapeutic application of CD19- and, increasingly, BCMA-targeting-engineered CAR-T cells have been approved and have yielded partly impressive results in hematologic malignancies. However, employing transgenic T cells for the treatment of solid tumors remains more troublesome, and numerous hurdles within the highly immunosuppressive tumor microenvironment (TME) need to be overcome to achieve tumor control. In this review, we focused on the challenges that these therapies must face on three different levels: infiltrating the tumor, exerting efficient antitumor activity, and overcoming T cell exhaustion and dysfunction. We aimed to discuss different options to pave the way for potent transgenic T cell-mediated tumor rejection by engineering either the TME or the transgenic T cell itself, which responds to the environment.