Spectrometer-based Fourier domain OCT is often used to measure blood flow velocities. Commonly, the axial component of the velocity is calculated from the phase difference of adjacent A-scans. While this result holds true for pure axial movement, a transversal component of the displacement will alter this simple relationship. We show theoretically and experimentally that because of the changing intensity of the illuminating beam on the moving particles the phase difference does not increase linearly with the velocity. Movements as small as 20% of the beam diameter during the integration time of the line detector will alter the observed phase shifts noticeably. For small angles between the transversal direction and the direction of movement the discrepancy between calculated and measured phase shift may be huge. High velocities at small angles will result in a limit for the phase shift smaller than π. A safe region, where the deviations to the linear relationship between axial velocity and phase shift are small, is specified.