Despite the fact that the local immunological microenvironment shapes the prognosis of colorectal cancer, immunotherapy has shown no benefit for the vast majority of colorectal cancer patients. A better understanding of the complex immunological interplay within the microenvironment is required. In this study, we utilized wet lab migration experiments and quantitative histological data of human colorectal cancer tissue samples (n ¼ 20) including tumor cells, lymphocytes, stroma, and necrosis to generate a multiagent spatial model. The resulting data accurately reflected a wide range of situations of successful and failed immune surveillance. Validation of simulated tissue outcomes on an independent set of human colorectal cancer specimens (n ¼ 37) revealed the model recapitulated the spatial layout typically found in human tumors. Stroma slowed down tumor growth in a lymphocyte-deprived environment but promoted immune escape in a lymphocyte-enriched environment. A subgroup of tumors with less stroma and high numbers of immune cells showed high rates of tumor control. These findings were validated using data from colorectal cancer patients (n ¼ 261). Low-density stroma and high lymphocyte levels showed increased overall survival (hazard ratio 0.322, P ¼ 0.0219) as compared with high stroma and high lymphocyte levels. To guide immunotherapy in colorectal cancer, simulation of immunotherapy in preestablished tumors showed that a complex landscape with optimal stroma permeabilization and immune cell activation is able to markedly increase therapy response in silico. These results can help guide the rational design of complex therapeutic interventions, which target the colorectal cancer microenvironment.