Rigorous and reliable alignment of narrow transceiver beams is a requisite for ensuring the highly directional transmission in millimeter-wave (mmWave) communications. Exhaustively testing these narrow beam pairs results in increased reference signal (RS) overhead, latency, and power consumption. In this paper, we propose a centralized multi-task learning (MTL) based beam prediction strategy that ensures a high success rate using measurements from a few site-specific probing beams identified via the proposed uniformly distributed beam relevance and beam significance (UDBRBS) criterion, thereby obviating the need for an exhaustive scan. Performance evaluation over 3rd Generation Partnership Project (3GPP) defined performance indicators demonstrates that the proposed method outperforms existing independent task learning (ITL) and single task learning (STL) beam prediction designs. We further argue that the proposed strategy is highly practical for implementation in fifth generation (5G)-Advanced and sixth generation (6G) communication systems.