Programmed death-ligand 1 (PD-L1) immunohistochemistry (IHC) is the most commonly used biomarker for immunotherapy response. However, quantification of PD-L1 status in pathology slides is challenging. Neither manual quantification nor a computer-based mimicking of manual readouts are perfectly reproducible, and the predictive performance of both approaches regarding immunotherapy response is limited. In this study, we developed a deep learning (DL) method to predict PD-L1 status directly from raw IHC image data, without explicit intermediary steps such as cell detection or pigment quantification. We trained the weakly supervised model on PD-L1-stained slides from the NSCLC-MSK cohort (N=233) and validated it on the pancancer-VHIO cohort (N=108). We also investigated the performance of the model to predict response to immune checkpoint inhibitors (ICIs) in terms of progression-free survival (PFS). In the pancancer-VHIO cohort, the performance was compared with Tumor Proportion Score (TPS) and Combined Positive Score (CPS). The DL-model showed good performance in predicting PD-L1 expression (TPS≥1%) in both NSCLC-MSK and pancancer-VHIO cohort (AUC 0.88±0.06 and 0.80±0.03, respectively). The predicted PD-L1 status showed an improved association with response to ICIs (hazard ratio (HR) 1.5[95%CI 1-2.3], p-value=0.049) compared to TPS (HR 1.4[0.96-2.2], p-value=0.082) and CPS (HR 1.2[0.79-1.9], p-value=0.386). Notably, our explainability analysis showed that the model does not just look at the amount of brown pigment in the IHC slides, but also considers morphological factors such as lymphocyte conglomerates. Overall, end-to-end weakly supervised DL shows potential for improving patient stratification for cancer immunotherapy by analyzing PD-L1 immunohistochemistry, holistically integrating morphology and PD-L1-staining intensity.