Deep learning (DL) has demonstrated high accuracy in ECG analysis but lacks in explainability. Although explanations can be estimated using explainable artificial intelligence, their causality has not yet been sufficiently investigated. We present a generalizable method for extensively validating the DL explanations’ causality by relating them to clinically relevant ECG characteristics. We applied xECGArch, combining a long-term and a short-term model, for atrial fibrillation (AF) detection in 1521 single-lead ECGs, achieving an accuracy of 96.3%. The explanations match the diagnostic criteria of AF regarding rhythm and morphology. While the short-term model emphasizes morphology features such as P and fibrillatory waves, the long-term model focuses on QRS complexes. Moreover, the long-term model explanations strongly correlate with rhythm (p< 0.001). For improved clinical interpretability, we introduce a fused representation (xFuseMap), highlighting relevant explanations for rhythm and morphology. We thus demonstrate an explainable and interpretable DL application with potential for providing diagnostic support.