Vibration measurements of tools are significant for machine modelling, error prone state identification and machine operation optimization. In the milling process, forced vibrations are the main factor resulting in shape deviations as well surface roughness of machined work pieces and depend on the spindle speed and tool properties. Numerical models for predicting the dynamic behavior of machine systems can help to reduce vibrations. For an optimization, the model parameters need to be evaluated by simultaneously measuring the vibration of the tool tip in force direction and its vertical direction under different rotational speeds with high spatio-temporal resolution. In this work, a non-contact, single-sensor system and a signal processing algorithm are presented for measuring the vibrational behavior inside of a CNC milling machine with a known force. It enables in situ, simultaneous bi-directional vibration measurements directly at the tool tip with measurement rates up to 50 kHz, a circumferential resolution below 230 μm and a displacement uncertainty down to 40 nm, at rotational speeds up to 300 Hz. The dynamic behavior parameters of the tool are evaluated depending on the measured vibration spectrum with relative uncertainties of 10^-2, enabling model optimizations. The measurements show a decrease of the natural frequency with increasing spindle speed. While the laser-based measurement principle does not bias the vibrational behavior it inherently guarantees orthogonality of the sensing axes, as well.