The wetting and drying ability of porous building materials plays a significant role in moisture transfer in the building envelope and thus influences the durability and energy efficiency of buildings. The drying characteristics of the material provide insight into the moisture transport feature over a wide moisture content range, and hence, the drying process can assist in improving the accuracy of moisture transport modelling for the heat, air and moisture (HAM) simulation and in comprehending the hygric behavior of buildings responding to environmental change. In this paper, the drying behaviors of calcium silicate and ceramic brick were experimentally and numerically investigated. The approaches to determine the drying rates of the first and second drying durations were first evaluated and proposed. The influence of the surface air flow velocity and sample dimension on the drying process was analyzed. The descriptive parameters to assess the drying rate were further discussed. The drying coefficient was shown to be able to describe the drying ability of porous building materials. Numerical simulation was conducted and compared with the measurement for a better understanding of the coupled heat and moisture transfer phenomena during the drying process.