Urban intersections are hotspots for crashes because they provide a location for several traffic streams and types of road users to cross. A main cause of crashes is the misinformation of drivers as they fail to sense relevant visual information. We aimed to analyze the gaze behavior of car drivers in a variety of intersection scenarios, bringing together the partial findings of previous research, and examine the interdependencies of the contributing factors to provide a database for driver modeling. In a driving simulator study with 59 participants, we varied intersection scenarios regarding drivers’ right of way (yield sign, green traffic light), intersection type (T junction, X intersection), surrounding traffic (none, irrelevant, relevant), and intended driving maneuver (left turn, right turn, going straight). A total of 25 intersection scenarios were presented in a within-subjects design to a control group and a group with a cognitive load task (counting back in numbers of two). Fixations were coded regarding defined areas of interest in the field of view and separated according to three segments of the intersection approach: 75–50 m, 50–25 m, and 25–0 m before entering the intersection. The results show that the effect of surrounding traffic, secondary task engagement, and the intended driving maneuver changed dramatically depending on the right of way of the driver. Surrounding traffic primarily affected gaze behavior in scenarios of ceding the right of way close to the intersection entry. The cognitive load task increased fixations on the road center especially in situations where the driver had the right of way, but less in situations of ceding the right of way. Interactions with the type of intersection were only apparent for different driving maneuvers. This study provides a detailed and comprehensive picture of drivers’ attentional processes when approaching intersections which is relevant for understanding and modeling of driver behavior in urban traffic.