Digitized histopathology slides contain a wealth of information, only a fraction of which is being used in clinical routine. Deep learning can extract subtle visual features from digitized slides and thus can infer clinically relevant endpoints from raw image data. While classification and regression methods are well established in this domain, end-to-end prediction of patient survival still remains a comparably novel approach. To account for different follow-up times and censored data, previous approaches have largely used discretized survival data. Here, we demonstrate and validate EE-Surv, a powerful yet algorithmically simple method to predict survival directly from whole slide images which we validate in colorectal and gastric cancer, two clinically relevant and markedly different tumor types. We experimentally show that this method yields a highly significant prediction of survival and enables explainability of predictions. This method is publicly available under an open-source license and can be applied to any type of disease.