One of the biggest challenges in the development of learning-driven automated driving technologies remains the handling of uncommon, rare events that may have not been encountered in training. Especially when training a model with real driving data, unusual situations, such as emergency brakings, may be underrepresented, resulting in a model that lacks robustness in rare events. This study focuses on car-following based on reinforcement learning and demonstrates that existing approaches, trained with real driving data, fail to handle safety–critical situations. Since collecting data representing all kinds of possible car-following events, including safety–critical situations, is challenging, we propose a training environment that harnesses stochastic processes to generate diverse and challenging scenarios. Our experiments show that training with real data can lead to models that collide in safety–critical situations, whereas the proposed model exhibits excellent performance and remains accident-free, comfortable, and string-stable even in extreme scenarios, such as full-braking by the leading vehicle. Its robustness is demonstrated by simulating car-following scenarios for various reward function parametrizations and a diverse range of artificial and real leader data that were not included in training and were qualitatively different from the learning data. We further show that conventional reward designs can encourage aggressive behavior when approaching other vehicles. Additionally, we compared the proposed model with classical car-following models and found it to achieve equal or superior results.