In addition to the traditional staging system in colorectal cancer (CRC), the Immunoscore® has been proposed to characterize the level of immune infiltration in tumor tissue and as a potential prognostic marker. The aim of this study was to examine and validate associations of an immune cell score analogous to the Immunoscore® with established molecular tumor markers and with CRC patient survival in a routine setting. Patients from a population-based cohort study with available CRC tumor tissue blocks were included in this analysis. CD3+ and CD8+ tumor infiltrating lymphocytes in the tumor center and invasive margin were determined in stained tumor tissue slides. Based on the T-cell density in each region, an immune cell score closely analogous to the concept of the Immunoscore® was calculated and tumors categorized into IS-low, IS-intermediate, or IS-high. Logistic regression models were used to assess associations between clinicopathological characteristics with the immune cell score, and Cox proportional hazards models to analyze associations with cancer-specific, relapse-free, and overall survival. From 1,535 patients with CRC, 411 (27%) had IS-high tumors. Microsatellite instability (MSI-high) was strongly associated with higher immune cell score levels (p < 0.001). Stage I-III patients with IS-high had better CRC-specific and relapse-free survival compared to patients with IS-low (hazard ratio [HR] = 0.42 [0.27-0.66] and HR = 0.45 [0.31-0.67], respectively). Patients with microsatellite stable (MSS) tumors and IS-high had better survival (HR CSS = 0.60 [0.42-0.88]) compared to MSS/IS-low patients. In this population-based cohort of CRC patients, the immune cell score was significantly associated with better patient survival. It was a similarly strong prognostic marker in patients with MSI-high tumors and in the larger group of patients with MSS tumors. Additionally, this study showed that it is possible to implement an analogous immune cell score approach and validate the Immunoscore® using open source software in an academic setting. Thus, the Immunoscore® could be useful to improve the traditional staging system in colon and rectal cancer used in clinical practice.