Next-generation wireless communication networks will benefit from beamforming gain to utilize higher bandwidths at millimeter wave (mmWave) and terahertz (THz) bands. For high directional gain, a beam management (BM) framework acquires and tracks optimal downlink and uplink beam pairs through exhaustive beam scan. However, for narrower beams at higher carrier frequencies this leads to a huge beam measurement overhead that negatively impacts the beam acquisition and tracking. Moreover, volatility of mmWave and THz channels, user random mobility patterns, and environmental changes further complicate the BM process. Consequently, machine learning (ML) algorithms that can identify and learn complex mobility patterns and track environmental dynamics have been identified as a remedy. In this article, we provide an overview of the existing ML-based mmWave/THz BM and beam tracking techniques. Especially, we highlight key characteristics of an optimal BM and tracking framework. By surveying the recent studies, we identify some open research challenges and provide our recommendations that can serve as a future direction for researchers in this area.