Large amplitude motions in molecular solids are responsible for anomalous electrical characteristics in amphidynamic crystals. We study the effect of orientational dipolar disorder on charge transfer excitons (CTEs) in a donor-acceptor amphidynamic co-crystal of perylene-tetrabromophthalic anhydride (-TBPA). This co-crystal offers long range positional order of alternating donor and acceptor molecular entities arranged in stacks. Further it has a phase transition at 250 K due to progressive freezing of the dipolar orientational motion in the TBPA permanent dipole. The optical absorption band and photoluminescence at the semiconductor gap consists of at least three energetically separated CTE resonances that cover a spectral range of more than 400 meV. As the temperature is lowered below the phase transition the CTE band remains broad, but shows a shift of ∼150 meV to higher energy. On the basis of optical reflectivity, photoluminescence, and model calculations we interpret the room temperature CTE band as a collection of resonances in which at least one the nearest neighbour dipoles to the CTE is flipped from their lowest energy configuration. This first comprehensive optical investigation on an amphidynamic co-crystal demonstrates the importance of understanding the role of permanent dipoles in the CTE photophysics of organic semiconductors.