In this study, we investigated the perceptual differences in the acceleration sound of a passenger car during rapid acceleration between drivers and passengers. Previous studies have addressed the perceptual differences in acceleration sounds according to nationality, driving situation, and the taste and past experiences of customers. It is expected that a perceptual difference will occur between the driver and passengers. However, perceptual differences between drivers and passengers have not yet been reported. Laboratory experiments were conducted to address this unclear issue. A subjective evaluation of affective attributes for acceleration sounds, such as annoyance, sportiness, acceleration feeling, and preference in driver and passenger situations, was carried out by employing a multimodal simulator. Participants were allowed to perform non-driving-related activities in the passenger role, whereas they were induced to concentrate only on the driving situation in the driver role. Experimental results have shown that there is a significant perceptual difference between drivers and passengers. It was found that the acceleration sound level for passengers should be reduced compared to the driver's seat to improve the passenger's auditory satisfaction.