We study mechanical vibrations in milling with non-uniform pitch and variable helix tools. The process is modeled by a periodic delay differential equation with distributed delay, which takes into account, for example, the nonlinear cutting force behavior, the effect of runout, and the exact delay distribution due to the unequally spaced flutes. We present a new method for the identification of the chatter stability lobes from the linearized system that is based on the multifrequency solution. We give detailed remarks on the truncation of the resulting infinite dimensional matrices and the efficient numerical implementation of the method. Cutting tests for steel milling with a customary end mill with non-uniform pitch and variable helix angle and a conventional end mill with uniform pitch and constant helix angle are performed. The numerical and experimental results coincide well. They reveal a significant increase of the limiting depth of cut for the variable helix tool compared to the conventional tool. Moreover, we show that in contrast to conventional tools, for non-uniform pitch and variable helix tools, an exact model with time-varying coefficients, nonlinear cutting force behavior, and runout is necessary for an accurate prediction of the stability lobes.