Radiomics has shown great potential for outcome prognosis and presents a promising approach for improving personalized cancer treatment. In radiomic analyses, features of different complexity are extracted from clinical imaging datasets, which are correlated to the endpoints of interest using machine-learning approaches. However, it is generally unclear if more complex features have a higher prognostic value and show a robust performance in external validation. Therefore, in this study, we developed and validated radiomic signatures for outcome prognosis after neoadjuvant radiochemotherapy in locally advanced rectal cancer (LARC) using computed tomography (CT) and T2-weighted magnetic resonance imaging (MRI) of two independent institutions (training/validation: 94/28 patients). For the prognosis of tumor response and freedom from distant metastases (FFDM), we used different imaging features extracted from the gross tumor volume: less complex morphological and first-order (MFO) features, more complex second-order texture (SOT) features, and both feature classes combined. Analyses were performed for both imaging modalities separately and combined. Performance was assessed by the area under the curve (AUC) and the concordance index (CI) for tumor response and FFDM, respectively. Overall, radiomic features showed prognostic value for both endpoints. Combining MFO and SOT features led to equal or higher performance in external validation compared to MFO and SOT features alone. The best results were observed after combining MRI and CT features (AUC = 0.76, CI = 0.65). In conclusion, promising biomarker signatures combining MRI and CT were developed for outcome prognosis in LARC. Further external validation is pending before potential clinical application.