Serrated end mills are widely used for rough machining of difficult-to-cut materials in the aerospace industry because of their reduced cutting forces and improved dynamic stability compared with conventional tools. However, stability prediction for serrated end mills remains challenging because of non-uniform chip thickness, multiple regenerative time delays, and the missed-cut effect. Existing stability models generally determine cutter engagement from nominal chip thickness and prescribed tool-workpiece kinematics, thereby neglecting the influence of forced-vibration-induced relative motion on instantaneous chip thickness and local cutter engagement.The fundamental advancement of this work is the incorporation of vibration-induced relative motion into the determination of instantaneous chip thickness, cutter engagement, missed cuts, and effective regenerative time delays. This leads to an improved stability model formulated as a delay differential equation with state-dependent delay, in which the delay is determined directly from the instantaneous full chip thickness. Newton–Raphson iteration based on the circular convolution operator is employed to obtain the converged forced periodic solutions, and the semi-discretization method is then applied to determine stability boundaries by linearizing the dynamics about these periodic solutions. The proposed model predicts substantially different stability limits from conventional models, particularly near machine-tool resonance, where vibration-induced chip-thickness modulation becomes significant at higher feed rates and lower radial immersion, resulting in pronounced variations in cutter engagement and effective time delays. The model is validated using results from the literature and milling experiments on steel with a sinusoidal serrated cutter having uniform helix and pitch. The experimental results confirm increases in the predicted stability boundary of up to 37% and 160% at feed rates of 0.1 and 0.25 mm/tooth, respectively, and demonstrate that serrated cutters retain their stabilizing effect even at large feed-to-serration-amplitude ratios, contrary to predictions of conventional stability models. The proposed framework enables more accurate prediction of the dynamic stability of serrated milling and can be extended to cutters with runout and non-uniform helix or pitch angles.