Spectral element methods on tetrahedra with symmetric collocation points can be accelerated by factorizing the discrete operators according to Hesthaven and Teng [SIAM J. Sci. Comput., 21 (2000), pp. 2352–2380]. While these authors focused on first-order conservation laws, the present paper provides an extension to second-order problems. Though factorization is easily accomplished for planar elements, difficulties arise from the presence of variable metric coefficients in curved tetrahedra. Two approaches are considered to cope with this peculiarity: (i) approximation of the metric terms by collocation projection, (ii) Gauss quadrature based on axisymmetric point sets. The first method achieves a separation of the metric terms such that the discrete operators can be reduced to factorizable standard matrices. As a consequence, the performance is comparable to the planar case, whereas the accuracy is limited by the projection step. The second approach maintains accuracy since all terms are evaluated individually in the quadrature points. Nonetheless, complete factorization is achieved by exploiting the symmetry in the quadrature points. Performance analysis shows that the curved element operator is less than three times as costly as the planar element counterpart.