Vibrations of milling tools, especially of slender end mills, are one of the main contributors to shape deviations of machined work pieces in the milling process. For milling machines, the vibration behaviour depends on the spindle speed and end mill properties. Numerical models can help to predict vibrational behaviour and reduce vibrations. This requires measuring the frequency response of the end mill. We propose a system for investigating the frequency response function inside of a milling machine under different rotational speeds. It consists of an automated impact unit generating vibrations with a known force and a novel Mach-Zehnder interferometer based sensor. The sensor enables simultaneous axial distance and lateral velocity measurement at fast rotating surfaces with submicron uncertainty and measurement rates up to 50 kHz. The bidirectional vibration frequencies are estimated by the signal-processing algorithm depending on the geometric relationship of the displacement in two directions.