Rotating instability (RI) can be regarded as a pre-stall signal because such phenomena are usually detected when a compressor operates at a near stall point. Since RI is closely relevant to aerodynamic, acoustic, and structural issues, its mechanism has attracted industrial concerns and in-depth analysis. In this paper, a low-speed axial single compressor rotor row is selected as the baseline research object. A circumferential casing groove is mounted over the shroud. The solid casing and the casing treated configurations were simulated using the zonal large eddy simulation model, and the simulations present remarkable predictions in capturing the averaged and the dynamic characteristics of the investigated compressor. By implementing a data-driven method called dynamic mode decomposition, the mode with the highest amplitude is proved to be correlated to the dominant RI frequency. The spatial and temporal property of the RI is visualized. The results show that the casing groove has limited influence on the averaged flow field but manipulates the dynamic features. The groove allows the RIs to shift from around 0.5 blade passing frequency to higher frequencies in the upstream region and effectively suppresses the unsteadiness in the zones under and downstream of the groove. The RIs are triggered by the flow interaction of the tip leakage flow at the front part of the blade. It can be depicted as pressure waves staggering with the rotor blade and propagating over the blade tip region at around half of the blade speed. With the existence of the casing groove, the dominant wavelength changes, and the fluctuation is damped under the groove, which finally modifies the properties of RIs.