Safety-critical vehicle-to-everything (V2X) services require stringent latency and reliability quality of service (QoS) under fast channel dynamics and dense traffic. In millimeter wave (mmWave) vehicular networks, analog beamforming mitigates path loss but enforces beam exclusivity, which couples multivehicle association decisions and complicates efficient handover. This paper proposes Handover Optimization in Vehicular Networks via Offline Reinforcement Learning (HONOR), an O-RAN-compliant handover xApp. Specifically, we formulate multi-vehicle handover as a Constrained Markov Decision Process (CMDP) subject to latency and reliability and handle the constraints via Lagrangian relaxation. To enforce one-to-one matching, an assignment-constrained factorized deep Q-learning network with validity masking is designed to obtain a feasible association via assignment-based action selection. Using a fixed dataset generated by O-RAN-integrated ns-3 simulations, the policy is learned offline and implemented as the HONOR xApp. Extensive evaluations with multiple random seeds show that HONOR reduces handover frequency by 18%-73% and suppresses ping-pong events by 41%- 81% relative to representative baselines, while improving latency and slightly increasing packet delivery reliability.