The tumor immune contexture plays a major role for the clinical outcome of patients. High densities of CD45RO⁺ T helper 1 cells and CD8⁺ T cells are associated with improved survival of patients with various cancer entities. In contrast, a higher frequency of tumor-infiltrating M2 macrophages is correlated with poor prognosis. Recent studies provide evidence that the tumor immune architecture also essentially contributes to the clinical efficacy of immune checkpoint inhibitor (CPI) therapy in patients. Pretreatment melanoma samples from patients who experienced a clinical response to anti-programmed cell death protein 1 (PD-1) treatment show higher densities of infiltrating CD8⁺ T cells compared to samples from patients that progressed during therapy. Anti-PD-1 therapy results in an increased density of tumor-infiltrating T lymphocytes in treatment responders. In addition, elevated frequencies of melanoma-infiltrating TCF7⁺CD8⁺ T cells are correlated with beneficial clinical outcome of anti-PD-1-treated patients. In contrast, a high density of tumor-infiltrating, dysfunctional PD-1⁺CD38^hi CD8⁺ cells in melanoma patients is associated with anti-PD-1 resistance. Such findings indicate that comprehensive tumor immune contexture profiling prior to and during CPI therapy may lead to the identification of underlying mechanisms for treatment response or resistance, and the design of improved immunotherapeutic strategies. Here, we focus on studies exploring the impact of intratumoral T and B cells at baseline on the clinical outcome of CPI-treated cancer patients. In addition, recent findings demonstrating the influence of CPIs on tumor-infiltrating lymphocytes are summarized.