In axial compressors, an unsteady flow phenomenon associated with fluctuating tip leakage flow can be observed at stable operating points with high aerodynamic loading close to stall, named rotating instability (RI). Within the frequency spectrum, e.g., from wall pressure measurements, RI is visualized as a hump lower than the blade passing frequency. Since RI indicates an increase in noise level and might cause blade vibration and other undesirable structural issues, it has been a hot-spot topic in the turbomachinery community over the last decades. In this paper, a study on RI is performed based on a single compressor rotor row of the Low Speed Research Compressor at Technische Universität Dresden. Firstly, setups with a solid casing and a casing groove mounted over the shroud are investigated based on the configuration with a blade tip clearance of 4.3flow coefficient) and the casing groove. By implementing dynamic mode decomposition on the transient simulation snapshots, the coherent flow structure corresponding to the dominant frequencies of RI is extracted and visualized as a wave-like flow feature located in the blade tip region. The RI is essentially equivalent to the superposition of periodic unsteady behaviors of the tip leakage vortex (TLV). Furthermore, RI is not detected in the configuration with a reduced blade tip clearance (1.3, as the tip flow becomes less unsteady. Based on the comprehensive analysis, two criteria for the occurrence of RI are concluded as follows: The tip leakage flow from one blade tip gap can influence the formation of the TLV originating from the adjacent blade, and the inherent unsteadiness of the tip flow is intensive enough.