We propose a reinforcement learning (RL) framework for energy-efficient control of cellular networks with fine-grained sector- and cell-level management. The approach dynamically adjusts activation states through a reward function that incorporates realistic power-amplifier (PA) efficiencies and user-connectivity constraints. By embedding empirically derived PA models based on measured hardware characteristics, the environment accurately captures energy use under synthetic mobility and handover conditions. Using Proximal Policy Optimization (PPO), the agent achieves up to 50.8% power reduction compared to a static baseline in our model, while maintaining user connectivity and rate satisfaction under the given assumptions. These results demonstrate the potential of hardware-aware RL for scalable energy optimization in next-generation networks.