In this paper, we investigate the pertinence of the angle of arrival (AoA) as a feature for robust physical layer authentication (PLA). While most of the existing approaches to PLA focus on amplitude-dependent features of the physical layer of communication channels, such as channel frequency response, channel impulse response, or received signal strength, the use of AoA in this domain has not yet been studied in depth, particularly regarding the ability to thwart spoofing (impersonation) attacks. In this work, we demonstrate that an impersonation attack targeting AoA-based PLA is only feasible under strict conditions on the attacker's location, which highlights the AoA's role as a strong feature for unspoofable PLA, especially when 2D AoA is employed. We extend previous works considering a single-antenna attacker to the case of a multiple-antenna attacker, and we develop a theoretical characterization of the conditions under which a successful impersonation attack can be mounted. Furthermore, we have performed extensive simulations in support of theoretical analyses, to validate the robustness of AoA-based PLA.