Convolutional Neural Networks (CNNs), such as VGG and ResNet, have been widely used for image classification for several years. Numerous explainable AI (XAI) algorithms, including Grad-CAM and XRAI, have been proposed to enhance interpretability in this domain. However, a persistent challenge lies in quantitatively comparing different XAI algorithms, variants of the same algorithm, or combinations of CNN models and XAI algorithms. In this work, we introduce XAIEV – a versatile framework for computationally evaluating the quality of saliency-map-based XAI algorithms. This framework includes (A) a benchmark dataset (traffic sign recognition) with a known ground truth and (B) a software toolbox designed to facilitate the evaluation pipeline. The pipeline consists of four steps: (1) model training, (2) applying XAI algorithms to generate weighted saliency maps, (3) generating new test images with varying percentages of “important” pixels removed or retained, and (4) statistically evaluating accuracy changes on these test images and comparison to the ground truth. Based on this statistical evaluation, we define an Accuracy-Sensitivity Quotient (ASQ) as a novel quality metric for XAI algorithms applied to image classification. Using the XAIEV framework, we compare various combinations of CNN architectures (“SimpleCNN” (custom model), VGG, ResNet, ConvNext) with multiple XAI algorithms (Grad-CAM, XRAI, LIME, PRISM). Our numerical results reveal that the performance of XAI algorithms is highly dependent on the underlying CNN model.