Although neoadjuvant radiochemotherapy (nRCTx) is an established oncological treatment in patients with advanced rectal cancer, little is known about its effects on the tumor microenvironment. Quantity and composition of tumor infiltrating lymphocytes (TILs) are known to influence patients' prognosis but nRCTx-induced modifications are still unclear. We determined the composition of the immune cell infiltrate in rectal cancer after nRCTx and its influence on tumor regression, local recurrence rate and survival. We investigated density and composition of tumor infiltrating CD3+ and CD8+ T-cells and the quantity and ratio of CD8+/GrzB+ T-cells to CD8+ T-cells in 130 rectal cancers after nRCTx compared to a cohort of 30 primarily resected rectal cancers. Furthermore, we analyzed 22 pretherapeutic rectal cancer biopsies, later treated with nRCTx and surgery to evaluate nRCTx-induced modifications of the tumor microenvironment. The total numbers of CD3+ and CD8+ T-cells in tumor stroma (p < 0.001) and tumor epithelium (p < 0.001 CD3; 0.002 CD8) were significantly lower in rectal cancers after nRCTx compared to primarily resected cases, while the ratio of CD8+/GrzB+ T-cells to CD8+ T-cells was significantly increased in the nRCTx cohort (p < 0.001). In multivariate analyses, CD8+/GrzB+ T-cells in the tumor stroma were significantly associated with high regression grade and a lower likelihood of local recurrence (p = 0.029). nRCTx modifies the tumor microenvironment of rectal cancer leading to a total decrease of TILs, but a relative increase in CD8+/GrzB+ T-cells in the tumor stroma. CD8+/GrzB+ T-cells may contribute to local tumor control and the better outcome.