Negative differential resistance (NDR) devices have emerged as promising building blocks for neuromorphic computing due to their inherent nonlinear and threshold-dependent electrical behavior. In particular, current-controlled NDR devices can exhibit abrupt switching, self-sustained oscillations, and volatility - key characteristics analogous to the spiking dynamics of biological neurons. The NDR is typically driven by thermal runaway effects, or by an insulator-to-metal transition, with materials such as VO2, NbOx, TaOx, and complex oxides like nikelates, cobaltites, or manganites. However, these systems often suffer from high forming and operating voltages, large Joule heating, limited device endurance, and no or little tunability. Here, we demonstrate a novel NDR device based on polycrystalline ErMnO3 comprising conducting orthorhombic and insulating hexagonal phases. Pt/ErMnO3/Pt memory devices exhibit unipolar, symmetric, forming-free threshold switching with a low threshold voltage of 2.1 V, a memory window of 1.1 V, and endurance over 2 & times; 104 cycles. Joule-heating-enhanced Poole-Frenkel conduction occurs in the orthorhombic phase, while the insulating hexagonal phase prevents excessive heating and breakdown. Tunability is achieved by controlling the polymorph ratio and the orthorhombic phase conductivity. ErMnO3 polymorphs thus offer a versatile platform to engineer the electrical characteristics of NDR devices for low-power, reliable neuromorphic applications.