Background: Decision-making processes may play a pivotal role in the etiology and maintenance of specific phobia. However, empirical evidence is limited. This study examined whether decision-making is only impaired in presence of fear-related stimuli or whether general impairments exist but are more pronounced in the presence of fear-related stimuli. Further, we examine which components of the decision-making process might be impaired. Methods: We examined a spider phobia group (SP, n = 109) relative to matched healthy controls (HC, n = 81) using a virtual decision game. To tap the approach-avoidance-conflict, either a fear-related version (using spiders) or a non-phobic version of the task was used in a between-subjects design to measure how the presence of fear-related or non-phobic stimuli was associated with optimal decision-making (collecting rewards). Based on drift diffusion modelling, underlying decision-making processes such as processing ability and cautiousness were investigated. Results: No clear evidence for general impairments of decision-making for SP participants relative to HC in the absence of fear-related stimuli was found, but a strong phobia-specific impairment when fear-related stimuli were present. These avoidant decisions were associated with a reduced ability to process the optimal choice option and increased cautiousness in the SP group. Conclusions: Decision-making processes in specific phobia are specifically impaired in the presence of fear-related stimuli, which might contribute to maladaptive, costly avoidance behavior.