Surrogate measures of safety (SMoS) play an important role in detecting traffic conflicts and in traffic safety assessment. However, the underlying assumptions of SMoS are different and a certain SMoS may be adequate/inadequate for different applications. A comprehensive approach to evaluate the validity and applicability of SMoS is lacking in the literature. This study proposes such a framework that supports evaluating SMoS in multiple dimensions. We apply the framework to gain insights into the characteristics of six widely-used SMoS for longitudinal maneuvers, i.e., Time to Collision (TTC), single-step Probabilistic Driving Risk Field (S-PDRF), Deceleration Rate to Avoid a Crash (DRAC), Potential Index for Collision with Urgent Deceleration (PICUD), Proactive Fuzzy Surrogate Safety Metric (PFS), and the Critical Fuzzy Surrogate Safety Metric (CFS). To ensure comparability, all measures are calibrated with the same risk detection criterion. Four performance indicators, i.e., Prediction Accuracy, Timeliness, Robustness, and Efficiency are computed for all six SMoS and validated using naturalistic driving data. The strengths and weaknesses of all six measures are compared and analyzed elaborately. A key result is that not a single SMoS performs well in all performance dimensions. S-PDRF performs best in terms of Robustness but consumes the most time for computation. TTC is the most efficient but performs poorly in terms of Timeliness and Robustness. The proposed evaluation approach and the derived insights can support SMoS selection in active vehicle safety system design and traffic safety assessment.