The properties of pavement materials strongly affect the service performance of the pavement. Although technical standards are commonly referenced to filter out unqualified materials, the method of deciding the optimal material remains unclear. This study proposes a decision-making framework based on the fuzzy comprehensive evaluation (FCE) and analytic hierarchy process (AHP). In this framework, several experiment-based indexes have been adopted to evaluate the service performance of pavement materials from different perspectives. Each index was quantitatively graded into five grades according to the fuzzy mathematics theories. The weights of the involved indexes were determined using AHP. The proposed decision-making framework was applied to selecting the optimal mineral filler from four types of mineral fillers for the pavement structure in a selected region. The proposed method combined the qualitative and quantitative factors and thus improved the credibility of the material applicability evaluation.