Currently, recovering the natural hydrological cycle and lowering urban flood risk are global challenges. A permeable pavement with void-rich pavement materials, such as porous asphalt (PA), is one feasible and effective measure. Due to the specific gradation, fillers can considerably influence the performance of PA mixtures. In this study, the effect of four different fillers (limestone, dolomite, rhyolite, and granodiorite) on the mechanical responses of PA mixtures was investigated by the numerical method. X-ray computer tomography (X-ray CT) and digital image processing (DIP) were applied to detect and reconstruct the microstructure of PA specimens. Finite-element (FE) analysis was conducted to simulate an indirect tensile test and by this means the mechanical responses of the PA mixtures (load-bearing capacity, von Mises stress, and creep strain) were computed and compared. The results show that the different fillers have significant influence on the mechanical responses of PA mixtures. Based on the computational results, the correlation between the mechanical responses of PA mixtures and filler properties was analyzed. In addition, the performance of PA mixtures with different fillers was ranked. Based on this ranking, it is possible to select an optimal filler to facilitate the improvement of permeable pavement design.