Road traffic noise is one of the most prominent sources of urban noise pollution. Recently, as a result of the electrification of vehicles, lower noise levels are expected in urban areas at speeds below 50 km/h. The commonly used physical descriptor, the A-weighted sound pressure level, does not sufficiently characterize the perceived annoyance of either combustion engine or electric passenger car pass-by noises. Psychoacoustical descriptors are advantageous for characterization purposes. The objective of this study was to evaluate the perceived annoyance caused by the noise from internal combustion engine, electric, and hybrid passenger cars using psychoacoustical properties. To achieve this objective, the following steps were conducted. First, the binaurally recorded sounds of 40 cars from different brands with different motorization were presented to the subjects who indicated the intensity of their perceived annoyance on a quasi-continuous scale. Second, the signal and psychoacoustical properties of the recorded sounds were analyzed. Third, a new annoyance index was developed, based on annoyance judgments and signal and psychoacoustical properties, to characterize the annoyance caused by pass-by noises. One of the novel aspects of this study is the consideration of not only the pass-by sounds of the internal combustion engine passenger cars, but also the pass-by sounds of hybrid and electric passenger cars. An acceleration from idle to a target speed of 50 km/h was selected as a traffic pass-by situation, which differs from those considered in previous studies. The results of the study show that psychoacoustical properties, such as loudness, tonality, roughness, and fluctuation strength, are very useful to characterize the annoyance perception, which is caused by single passenger car pass-by sounds in the above mentioned traffic situation. The developed index, which is a weighted combination of chosen psychoacoustical properties, can be very useful for traffic planning and traffic noise prevention measures.