We report the first observation of negative photoconductance (NPC) in n- and p-doped Si nanowire field-effect transistors (FETs) and demonstrate the strong influence of doping concentrations on the nonconventional optical switching of the devices. Furthermore, we show that the NPC of Si nanowire FETs is dependent on the wavelength of visible light due to the phonon-assisted excitation to multiple conduction bands with different band gap energies that would be a distinct optoelectronic property of indirect band gap semiconductor. We attribute the main driving force of NPC in Si nanowire FETs to the photogenerated hot electrons trapping by dopants ions and interfacial states. Finally, comparing back- and top-gate modulation, we derive the mechanisms of the transition between negative and positive photoconductance regimes in nanowire devices. The transition is decided by the competition between the light-induced interfacial trapping and the recombination of mobile carriers, which is dependent on the light intensity and the doping concentration.